Hi, I'm struggling with a simple update of PuppetDB since a couple of days, without finding the problem. I have 4 PuppetServers running Puppetserver 6.9 (puppetserver-6.9.0-1.el7.noarch). One has the CA role, the 3 others are simple masters. I have one dedicated PuppetDB server running puppetdb-6.5.0-1.
Everything is working like a charm since a couple of years. It was updated from Puppet 3, 4 and 6 without a glitch. Everything is running on CentOS 7. Now, when I want to update PuppetDb from 6.5 to 6.9, nothing works anymore. All nodes are complaining with these messages : Warning: Unable to fetch my node definition, but the agent run will continue: Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for vmlabybr06.staging.rsvgnw.local: Failed to find facts from PuppetDB at vmprdpuppet41.rsvgnw.local:8140: Failed to execute '/pdb/query/v4/nodes/vmlabybr06.staging.rsvgnw.local/facts' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=5da252cdae0fc1737726e9ace846d74856395703&version=5&certname=vmlabybr06.staging.rsvgnw.local&command=replace_facts&producer-timestamp=2020-04-09T13:15:44.382Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run In the server log I get this : 2020-04-09T15:22:45.169+02:00 WARN [qtp1002336767-143] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request javax.net.ssl.SSLException: Received fatal alert: handshake_failure at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271) at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316) at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.lang.Thread.run(Thread.java:748) 2020-04-09T15:22:45.171+02:00 WARN [qtp1002336767-143] [puppetserver] Puppet Error connecting to vmctldeploy20.rsvgnw.local on 8081 at route /pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list 2020-04-09T15:22:45.172+02:00 ERROR [qtp1002336767-143] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081 I have checked a few things : - Updated puppetdb-termini on the puppet-master from 6.5 to 6.9 (no change) - added "verify_client_certificate = false" to /etc/puppetlabs/puppet/puppetdb.conf on the masters (no change) - added full certs list to PuppetDB server /etc/puppetlabs/puppetdb/ssl/public.pem I've read there has been a change liked to SSL in the PuppetDB 6.6 CHANGELOG. Here is what happens when I try to connect with openssl for troubleshooting, to PuppetDB 6.5 openssl s_client -host puppetdb -port 8081 -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = Puppet CA: vmctldeploy10.rsvgnw.local verify return:1 depth=0 CN = vmctldeploy20.rsvgnw.local verify return:1 140503727654720:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42 --- Certificate chain 0 s:CN = vmctldeploy20.rsvgnw.local i:CN = Puppet CA: vmctldeploy10.rsvgnw.local --- Server certificate -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- subject=CN = vmctldeploy20.rsvgnw.local issuer=CN = Puppet CA: vmctldeploy10.rsvgnw.local --- Acceptable client certificate CA names CN = Puppet CA: vmctldeploy10.rsvgnw.local Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA256 Peer signature type: RSA Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2213 bytes and written 455 bytes Verification: OK --- The only way to go back is doing a full "revert to snaphot", as the db is migrated between 6.5 and 6.9 Any advise welcome ! Cheers Yvan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/33e92b3d-84d0-42ce-87ee-d958b8cf78d1%40googlegroups.com.
