Tuesday, December 31, 2019

Puppet node doesn't appear in the console.


PROBLEM:
Node doesn't appear in the console.
Running the agent throws this error:
[root@<managed_node> ~]# puppet agent -t
Info: Not using expired certificate for ca from cache; expired at 2019-10-07 16:57:13 UTC
Info: Not using expired certificate for ca from cache; expired at 2019-10-07 16:57:13 UTC
. . .
SOLUTION:
rm /etc/puppetlabs/puppet/ssl/certs/ca.pem
puppet agent -t
EOS

Thursday, November 28, 2019

Upgrading to Puppet Enterprise Master of Masters to 5.5.17 (2018.1.11), the Windows clients can't find cmd.exe


PROBLEM:
Upgrading to Puppet Enterprise Master of Masters to 5.5.17 (2018.1.11), the Windows clients can't find cmd.exe
PS C:\Windows\system32> puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for ntcfsxxaps0147.usda.net
Info: Applying configuration version '1574901148'
Error: Could not find command 'C:\Users\pmcbrj01\WINDOWS\system32\cmd.exe'
Error: /Stage[main]/Puppet_agent::Install::Windows/Exec[install_puppet.ps1]/returns: change from 'notrun' to ['0'] faile
d: Could not find command 'C:\Users\pmcbrj01\WINDOWS\system32\cmd.exe' (corrective)
Notice: /Stage[main]/Puppet_agent::Install::Windows/Exec[fix inheritable SYSTEM perms]: Dependency Exec[install_puppet.p
s1] has failures: true
Warning: /Stage[main]/Puppet_agent::Install::Windows/Exec[fix inheritable SYSTEM perms]: Skipping because of failed depe
ndencies
Notice: Applied catalog in 0.54 seconds
PS C:\Windows\system32> puppet agent --version
5.5.16
SOULTION:
Wait a little while to let the agent upgrade
PS C:\Windows\system32> puppet agent --version
5.5.17
…then try again
PS C:\Windows\system32> puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for ntcfsxxaps0147.usda.net
Info: Applying configuration version '1574902446'
Notice: Applied catalog in 0.56 seconds


Wednesday, October 30, 2019

PROBLEM:
Client nodes have a down level Puppet yum repository, i.e. it should be 2018.1.9 in this case
[root@yum.repos.d]# cat pc_repo.repo
[pc_repo]
name=Puppet Labs puppet5 Repository
baseurl=https://puppet:8140/packages/2018.1.8/el-7-x86_64
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
  file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppet
sslcacert=/etc/puppetlabs/puppet/ssl/certs/ca.pem
sslclientcert=/etc/puppetlabs/puppet/ssl/certs/clientnode.it.com.pem
sslclientkey=/etc/puppetlabs/puppet/ssl/private_keys/clientnode.it.com.pem
[root@ yum.repos.d]#

SOLUTION:
Create a group (for example 'Agent_Maintain')  Assign the puppet_agent class in the Configuration tab.
In the same tab, set the class' varaible:    "package_version": "5.5.6"
Pin the node.
Run the agent.
EOS

Saturday, September 28, 2019

PE migration, database restore fails with postgres errors

PROBLEM:
2019-09-28:13:30:52-database_restore.pm135.sh: .. Restoring database: pe-activity
file=/fslink/sysinfra/puppet/2018.1.9/backup/pm135/database//pe-activity_201909232245.bin
jobs=2
pg_restore: connecting to database for restore
pg_restore: dropping DATABASE pe-activity
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3099; 1262 16399 DATABASE pe-activity pe-postgres
pg_restore: [archiver (db)] could not execute query: ERROR:  database "pe-activity" is being accessed by other users
DETAIL:  There are 10 other sessions using the database.
    Command was: DROP DATABASE "pe-activity";

pg_restore: processing item 3097 ENCODING ENCODING
pg_restore: processing item 3098 STDSTRINGS STDSTRINGS
pg_restore: processing item 3099 DATABASE pe-activity
pg_restore: creating DATABASE "pe-activity"
pg_restore: [archiver (db)] could not execute query: ERROR:  database "pe-activity" already exists
    Command was: CREATE DATABASE "pe-activity" WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8...
pg_restore: connecting to new database "pe-activity"
pg_restore: connecting to database "pe-activity" as user "pe-postgres"
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3100 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3102 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: processing item 190 FUNCTION backup_activity_events(timestamp with time zone, text)
pg_restore: creating FUNCTION "public.backup_activity_events(timestamp with time zone, text)"
pg_restore: [archiver (db)] Error from TOC entry 190; 1255 17885 FUNCTION backup_activity_events(timestamp with time zone, text) pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  function "backup_activity_events" already exists with same argument types
    Command was: CREATE FUNCTION backup_activity_events(threshold timestamp with time zone, filename text) RETURNS void
    LANGUAGE plpgsql
...
pg_restore: processing item 188 TABLE event_commits
pg_restore: creating TABLE "public.event_commits"
pg_restore: [archiver (db)] Error from TOC entry 188; 1259 17711 TABLE event_commits pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "event_commits" already exists
    Command was: CREATE TABLE event_commits (
    id character(40) NOT NULL,
    service_id text NOT NULL,
    subject_digest_id text NOT NUL...
pg_restore: processing item 189 TABLE events
pg_restore: creating TABLE "public.events"
pg_restore: [archiver (db)] Error from TOC entry 189; 1259 17800 TABLE events pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "events" already exists
    Command was: CREATE TABLE events (
    id character(40) NOT NULL,
    commit_id text NOT NULL,
    action_type text NOT NULL,
    action_...
pg_restore: processing item 187 TABLE object_digests
pg_restore: creating TABLE "public.object_digests"
pg_restore: [archiver (db)] Error from TOC entry 187; 1259 17660 TABLE object_digests pe-activity
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "object_digests" already exists
    Command was: CREATE TABLE object_digests (
    id character(40) NOT NULL,
    object_type text NOT NULL,
    object_id text NOT NULL,
   ...
pg_restore: processing item 185 TABLE schema_migrations
<snip>

SOLTUION:
Cycle all the Puppet operating system services, then try importing the pe-activity database again.
sv=pxp-agent;                 echo .. $sv; puppet resource service $sv ensure=stopped
sv=puppet;                    echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-nginx;                  echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-console-services;       echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-puppetserver;           echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-puppetdb;               echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-orchestration-services; echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-postgresql              echo .. $sv; puppet resource service $sv ensure=stopped
sv=mcollective;               echo .. $sv; puppet resource service $sv ensure=stopped
sv=pe-activemq;               echo .. $sv; puppet resource service $sv ensure=stopped
ps -ef | grep ^[0-9]
pids=$(ps -ef | grep [p]ostgresql | awk '{print $2}')
[[ -z $pids ]] || kill $pids

Wednesday, September 25, 2019

Puppet agent throws: ...Unknown resource type: 'anchor' (file: ...

PROBLEM:
puppet agent -t throws this error:
RDBMS [root@rdb1210 ~]# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'r10k', switching agent to 'r10k'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'anchor' (file: /etc/puppetlabs/code/environments/r10k/site/profile/manifests/linode3/linode3.pp, line: 2, column: 4) on node rdb1210.lpcnextlight.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

SOLUTION:
On the Puppet Master, install puppetlabs-stdlib.  For example:
puppet module install puppetlabs-stdlib
puppet agent -t

Adding it to the Puppetfile also works.

Then on the node, try running the agent again:  
puppet agent -t

Wednesday, April 3, 2019

Troubleshoot hiera with "puppet lookup"


One of the challenging things with puppet is troubleshooting how it sets environment variables in its flavor of ruby.  With its tool (hiera) it reads a set of yaml flat files with key-value pairs.  The challengs is figuring out which file from the remote client, and what debug tool is the most recent.

The "puppet lookup" tool is the latest, and is an improvement.  It allows the Puppet Master act like it was reading like the client node with the --node option.

In this case, running the agent with "puppet agent -t" was throwing errors that profile::apache::package_ensure  was an undefined varaible of the apache module from the puppet forge.  We set it in the yaml file, captured the output from "puppet lookup", and compared it to a working client (node.)


#root$  cat /etc/puppetlabs/code/environments/production/hieradata/node/target1.mydomain.com.yaml

---

   profile::apache::package_ensure: 'present'



#root$  puppet lookup profile::apache::package_ensure --node target1.mydomain.com --explain  > /tmp/target1.txt  2&>1

#root$  puppet lookup profile::apache::package_ensure --node target2.mydomain.com --explain  > /tmp/target2.txt  2&>1

#root$  diff /tmp/target1.txt /tmp/target2.txt

(The file that dictates the order yaml files are read for the "production" environment is /etc/puppetlabs/code/environments/production/hiera.yaml, but that is dictated by /etc/puppetlabs/puppet/hiera.yaml.)

These days, "puppet lookup" is the way to go.