One of the challenging things with puppet is troubleshooting how it sets environment variables in its flavor of ruby. With its tool (hiera) it reads a set of yaml flat files with key-value pairs. The challengs is figuring out which file from the remote client, and what debug tool is the most recent.
The "puppet lookup" tool is the latest, and is an improvement. It allows the Puppet Master act like it was reading like the client node with the --node option.
In this case, running the agent with "puppet agent -t" was throwing errors that profile::apache::package_ensure was an undefined varaible of the apache module from the puppet forge. We set it in the yaml file, captured the output from "puppet lookup", and compared it to a working client (node.)
#root$ cat /etc/puppetlabs/code/environments/production/hieradata/node/target1.mydomain.com.yaml
---
profile::apache::package_ensure: 'present'
#root$ puppet lookup profile::apache::package_ensure --node target1.mydomain.com --explain > /tmp/target1.txt 2&>1
#root$ puppet lookup profile::apache::package_ensure --node target2.mydomain.com --explain > /tmp/target2.txt 2&>1
#root$ diff /tmp/target1.txt /tmp/target2.txt
(The file that dictates the order yaml files are read for the "production" environment is /etc/puppetlabs/code/environments/production/hiera.yaml, but that is dictated by /etc/puppetlabs/puppet/hiera.yaml.)
These days, "puppet lookup" is the way to go.
