Puppet console can't run tasks from the console

PROBLEM:
Running a task from the console returns:
Failed - error...
Error: fsxopsx1031.wrk.fs.usda.gov is not connected to the PCP broker

On a broken node, run:
/opt/puppetlabs/puppet/bin/pxp-agent --foreground --loglevel debug
tail /var/log/puppetlabs/pxp-agent/pxp-agent.log

It returns:
2020-01-03 11:48:34.877539 INFO  puppetlabs.pxp_agent.main:189 - pxp-agent 1.9.11 started at debug level
2020-01-03 11:48:34.877742 ERROR puppetlabs.pxp_agent.main:208 - Fatal configuration error: broker-ws-uri or broker-ws-uris must be defined; cannot start pxp-agent

SOLUTION:
Same solution as last post, repeated here:

Documentation says to set master_uris and pcp_broker_list for the PE Agent and PE Infrastructure Agent groups in the console (https://puppet.com/docs/pe/2018.1/installing_compile_masters.html).  For example:

Change the PXP agent to connect directly to the MoM, not the compile masters' loadbalancer.
Classification --> PE Infrastructure --> PE Agent --> Configuration tab
Class: puppet_enterprise::profile::agent
master_uris = ["https://<MoM_FQDN>/"]
pcp_broker_list = ["<MoM_FQDN>:8142"]

However, our experience is that removing master_uris and pcp_broker had a much higher success rate.

After removing them, run the agent on the managed node.  Successful connections should start appearing in 

tail -F /var/log/puppetlabs//pxp-agent/pxp-agent.log

Also, the new master_uris and pcp_broker_list values should appear in the pxp-agent.conf file that the MoM manages remotely on the managed node.  Look on the managed node:

cat /etc/puppetlabs/pxp-agent/pxp-agent.conf | python -m json.tool

EOS

Can't run remote agents from Puppet console. Also shuts down soon after starting.

PROBLEM:
Console won't run the agent saying, "Run Puppet has been disabled because Node Manager cannot connect to <fqdn>".
Also 'puppet job' can't run anything on any nodes from the MoM command line.

Tried:
Turn on debugging in activemq.

cp -p /etc/puppetlabs/activemq/log4j.properties /etc/puppetlabs/activemq/log4j.properties.orig

vim /etc/puppetlabs/activemq/log4j.properties

Comment this line:          #log4j.rootLogger=INFO, console, logfile

Uncomment this line:     log4j.rootLogger=DEBUG, logfile, console

Bounce the service:

sv=pe-activemq;               echo == $sv; puppet resource service $sv ensure=stopped

sv=pe-activemq;               echo == $sv; puppet resource service $sv ensure=running

The log rotates, so use -F

tail -F /var/log/puppetlabs/activemq/activemq.log

Certificate expiration messages started appearing:

2019-12-31T08:44:01.440-06:00 | WARN | Transport Connection to: tcp://<node_ip>:44166 failed: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: ssl:///<node_ip>:44166

Puppet support says to disable MCollective if we're not using it:

https://puppet.com/docs/pe/2018.1/disabling_mcollective_nodes.html

Trying to rekey Orchestration on the MoM:

mv /etc/puppetlabs/orchestration-services/ssl/<files> /tmp

cp <puppet>/ssl <orchestration

chown -R pe-orchestration-services:pe-orchestration-services /etc/puppetlabs/orchestraion-services/ssl

CAUSE:
The real cause was the PXP agent.  On one of the failing managed nodes, the PXP agent was throwing errors that it couldn't connect to wss://<compile_master_load_balancer>:8140/pcp2/agent

SOLUTION:
Documentation says to set master_uris and pcp_broker_list for the PE Agent and PE Infrastructure Agent groups in the console (https://puppet.com/docs/pe/2018.1/installing_compile_masters.html).  For example:

Change the PXP agent to connect directly to the MoM, not the compile masters' loadbalancer.
Classification --> PE Infrastructure --> PE Agent --> Configuration tab
Class: puppet_enterprise::profile::agent
master_uris = ["https://<MoM_FQDN>/"]
pcp_broker_list = ["<MoM_FQDN>:8142"]

However, our experience is that removing master_uris and pcp_broker had a much higher success rate.

After removing them, run the agent on the managed node.  Successful connections should start appearing in 

tail -F /var/log/puppetlabs//pxp-agent/pxp-agent.log

Also, the new master_uris and pcp_broker_list values should appear in the pxp-agent.conf file that the MoM manages remotely on the managed node.  Look on the managed node:

cat /etc/puppetlabs/pxp-agent/pxp-agent.conf | python -m json.tool

EOS

Puppet node doesn't appear in the console.

PROBLEM:

Node doesn't appear in the console.

Running the agent throws this error:

[root@<managed_node> ~]# puppet agent -t

Info: Not using expired certificate for ca from cache; expired at 2019-10-07 16:57:13 UTC

Info: Not using expired certificate for ca from cache; expired at 2019-10-07 16:57:13 UTC

. . .

SOLUTION:

rm /etc/puppetlabs/puppet/ssl/certs/ca.pem

puppet agent -t

EOS

Upgrading to Puppet Enterprise Master of Masters to 5.5.17 (2018.1.11), the Windows clients can't find cmd.exe

PROBLEM:

Upgrading to Puppet Enterprise Master of Masters to 5.5.17 (2018.1.11), the Windows clients can't find cmd.exe

PS C:\Windows\system32> puppet agent -t

Info: Using configured environment 'production'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

Info: Loading facts

Info: Caching catalog for ntcfsxxaps0147.usda.net

Info: Applying configuration version '1574901148'

Error: Could not find command 'C:\Users\pmcbrj01\WINDOWS\system32\cmd.exe'

Error: /Stage[main]/Puppet_agent::Install::Windows/Exec[install_puppet.ps1]/returns: change from 'notrun' to ['0'] faile

d: Could not find command 'C:\Users\pmcbrj01\WINDOWS\system32\cmd.exe' (corrective)

Notice: /Stage[main]/Puppet_agent::Install::Windows/Exec[fix inheritable SYSTEM perms]: Dependency Exec[install_puppet.p

s1] has failures: true

Warning: /Stage[main]/Puppet_agent::Install::Windows/Exec[fix inheritable SYSTEM perms]: Skipping because of failed depe

ndencies

Notice: Applied catalog in 0.54 seconds

PS C:\Windows\system32> puppet agent --version

5.5.16

SOULTION:

Wait a little while to let the agent upgrade

PS C:\Windows\system32> puppet agent --version

5.5.17

…then try again

PS C:\Windows\system32> puppet agent -t

Info: Using configured environment 'production'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

Info: Loading facts

Info: Caching catalog for ntcfsxxaps0147.usda.net

Info: Applying configuration version '1574902446'

Notice: Applied catalog in 0.56 seconds

PROBLEM:
Client nodes have a down level Puppet yum repository, i.e. it should be 2018.1.9 in this case
[root@yum.repos.d]# cat pc_repo.repo
[pc_repo]
name=Puppet Labs puppet5 Repository
baseurl=https://puppet:8140/packages/2018.1.8/el-7-x86_64
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
  file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppet
sslcacert=/etc/puppetlabs/puppet/ssl/certs/ca.pem
sslclientcert=/etc/puppetlabs/puppet/ssl/certs/clientnode.it.com.pem
sslclientkey=/etc/puppetlabs/puppet/ssl/private_keys/clientnode.it.com.pem
[root@ yum.repos.d]#

SOLUTION:
Create a group (for example 'Agent_Maintain')  Assign the puppet_agent class in the Configuration tab.
In the same tab, set the class' varaible:    "package_version": "5.5.6"
Pin the node.
Run the agent.
EOS

PE migration, database restore fails with postgres errors

PROBLEM:
2019-09-28:13:30:52-database_restore.pm135.sh: .. Restoring database: pe-activity
file=/fslink/sysinfra/puppet/2018.1.9/backup/pm135/database//pe-activity_201909232245.bin
jobs=2
pg_restore: connecting to database for restore
pg_restore: dropping DATABASE pe-activity
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3099; 1262 16399 DATABASE pe-activity pe-postgres
pg_restore: [archiver (db)] could not execute query: ERROR:  database "pe-activity" is being accessed by other users
DETAIL:  There are 10 other sessions using the database.
    Command was: DROP DATABASE "pe-activity";

pg_restore: processing item 3097 ENCODING ENCODING
pg_restore: processing item 3098 STDSTRINGS STDSTRINGS
pg_restore: processing item 3099 DATABASE pe-activity
pg_restore: creating DATABASE "pe-activity"
pg_restore: [archiver (db)] could not execute query: ERROR:  database "pe-activity" already exists
    Command was: CREATE DATABASE "pe-activity" WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8...
pg_restore: connecting to new database "pe-activity"
pg_restore: connecting to database "pe-activity" as user "pe-postgres"
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3100 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3102 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: processing item 190 FUNCTION backup_activity_events(timestamp with time zone, text)
pg_restore: creating FUNCTION "public.backup_activity_events(timestamp with time zone, text)"
pg_restore: [archiver (db)] Error from TOC entry 190; 1255 17885 FUNCTION backup_activity_events(timestamp with time zone, text) pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  function "backup_activity_events" already exists with same argument types
    Command was: CREATE FUNCTION backup_activity_events(threshold timestamp with time zone, filename text) RETURNS void
    LANGUAGE plpgsql
...
pg_restore: processing item 188 TABLE event_commits
pg_restore: creating TABLE "public.event_commits"
pg_restore: [archiver (db)] Error from TOC entry 188; 1259 17711 TABLE event_commits pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "event_commits" already exists
    Command was: CREATE TABLE event_commits (
    id character(40) NOT NULL,
    service_id text NOT NULL,
    subject_digest_id text NOT NUL...
pg_restore: processing item 189 TABLE events
pg_restore: creating TABLE "public.events"
pg_restore: [archiver (db)] Error from TOC entry 189; 1259 17800 TABLE events pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "events" already exists
    Command was: CREATE TABLE events (
    id character(40) NOT NULL,
    commit_id text NOT NULL,
    action_type text NOT NULL,
    action_...
pg_restore: processing item 187 TABLE object_digests
pg_restore: creating TABLE "public.object_digests"
pg_restore: [archiver (db)] Error from TOC entry 187; 1259 17660 TABLE object_digests pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "object_digests" already exists
    Command was: CREATE TABLE object_digests (
    id character(40) NOT NULL,
    object_type text NOT NULL,
    object_id text NOT NULL,
   ...
pg_restore: processing item 185 TABLE schema_migrations
<snip>

SOLTUION:
Cycle all the Puppet operating system services, then try importing the pe-activity database again.
sv=pxp-agent;                 echo .. $sv; puppet resource service $sv ensure=stopped
sv=puppet;                    echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-nginx;                  echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-console-services;       echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-puppetserver;           echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-puppetdb;               echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-orchestration-services; echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-postgresql              echo .. $sv; puppet resource service $sv ensure=stopped
sv=mcollective;               echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-activemq;               echo .. $sv; puppet resource service $sv ensure=stopped
ps -ef | grep ^[0-9]
pids=$(ps -ef | grep [p]ostgresql | awk '{print $2}')
[[ -z $pids ]] || kill $pids

Puppet agent throws: ...Unknown resource type: 'anchor' (file: ...

PROBLEM:
puppet agent -t throws this error:
RDBMS [root@rdb1210 ~]# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'r10k', switching agent to 'r10k'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'anchor' (file: /etc/puppetlabs/code/environments/r10k/site/profile/manifests/linode3/linode3.pp, line: 2, column: 4) on node rdb1210.lpcnextlight.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

SOLUTION:

On the Puppet Master, install puppetlabs-stdlib.  For example:

puppet module install puppetlabs-stdlib

puppet agent -t

Adding it to the Puppetfile also works.

Then on the node, try running the agent again:  

puppet agent -t