Three machines in the production environment had some hardware issues and were decommissioned. The infrastructure team has reinstalled them and gave them the same hostnames and IP addresses. The aim is to run Puppet on these systems so these can be commissioned again.
Attempt
1) The old Puppet certificates were removed from the Puppetmaster by issuing the following commands:
puppet cert revoke grb16.company.compuppet cert clean grb16.company.com2) Once the old certificate was removed, a new certificate request was created by issuing the following command from one of the reinstalled nodes:
[root@grb16 ~]# puppet agent -tInfo: csr_attributes file loading from /etc/puppet/csr_attributes.yamlInfo: Creating a new SSL certificate request for grb16.company.comInfo: Certificate Request fingerprint (SHA256): 6F:2D:1D:71:67:18:99:86:2C:22:A1:14:80:55:34:35:FD:20:88:1F:36:ED:A7:7B:2A:12:09:4D:F8:EC:BF:6DExiting; no certificate found and waitforcert is disabled[root@grb16 ~]#3) Once the certificate request was visible on the Puppetmaster, the following command was issued to sign the certificate request:
[root@foreman ~]# puppet cert sign grb16.company.comNotice: Signed certificate request for grb16.company.comNotice: Removing file Puppet::SSL::CertificateRequest grb16.company.com at '/var/lib/puppet/ssl/ca/requests/grb16.company.com.pem'[root@foreman ~]# Problem
Once the certificate request has been signed and a Puppet run has been started the following error is thrown:
[root@grb16 ~]# puppet agent -tInfo: Caching certificate for grb16.company.comError: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Exiting; failed to retrieve certificate and waitforcert is disabled[root@grb16 ~]# Running Puppet for the second time results in:
[root@grb16 ~]# puppet agent -tWarning: Unable to fetch my node definition, but the agent run will continue:Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Info: Retrieving pluginfactsError: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://foreman.company.com/pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Wrapped exception:SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Info: Retrieving pluginError: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://foreman.company.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Wrapped exception:SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]Warning: Not using cache on failed catalogError: Could not retrieve catalog; skipping runError: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com][root@grb16 ~]# Analysis
In order to solve the issue, the error message were investigated and it looks like that the problem is SSL or Puppet related. Perhaps one of these packages has been installed incorrectly or a wrong version has been installed on the reinstalled node.
Puppet
[root@grb16 ~]# yum list installed |grep puppetfacter.x86_64 1:2.3.0-1.el6 @puppetlabs_6_products hiera.noarch 1.3.4-1.el6 @puppetlabs_6_products puppet.noarch 3.7.3-1.el6 @puppetlabs_6_products puppetlabs-release.noarch 6-11 @puppetlabs_6_products ruby-augeas.x86_64 0.4.1-3.el6 @puppetlabs_6_deps ruby-shadow.x86_64 1:2.2.0-2.el6 @puppetlabs_6_deps rubygem-json.x86_64 1.5.5-3.el6 @puppetlabs_6_deps SSL
[root@grb16 ~]# yum list installed |grep sslnss_compat_ossl.x86_64 0.9.6-1.el6 @anaconda-CentOS-201410241409.x86_64/6.6openssl.x86_64 1.0.1e-30.el6_6.4openssl-devel.x86_64 1.0.1e-30.el6_6.4[root@grb16 ~]# No discrepancies were found between the SSL and Puppet packages that are installed on various servers. The systems that have not been decommissioned or reinstalled are still able to run Puppet. The issue is restricted to the reinstalled server. Note that Puppet has not been run on the other two reinstalled servers. What is causing this issue and how to solve it?