On Wed, Nov 27, 2013 at 9:55 AM, Ken Barber <[email protected]> wrote: >> I run all my Puppetised servers on CentOS 6.4. Overnight there were a load >> of updates for CentOS including an update to openssl-1.0.1e-15.el6. Since >> installing the updates, PuppetDB is no longer working and seems to be having >> troubles with SSL. >> >> All my puppet nodes show: >> >> Error: Could not retrieve catalog from remote server: Error 400 on SERVER: >> Failed to submit 'replace facts' command for >> radius-dev.nomadic-core.bris.ac.uk to PuppetDB at >> puppetdb.resnet.bris.ac.uk:8081: Connection refused - connect(2) >> >> The PuppetDB server shows: >> >> 2013-11-27 12:09:58,347 WARN [qtp1710594959-45] [io.nio] >> javax.net.ssl.SSLException: Inbound closed before receiving peer's >> close_notify: possible truncation attack? >> >> Has anyone else had this problem? Any tips? I recreated the PuppetDB certs >> but this didn't help. > > This all sounds pretty serious, but something isn't quite right here > with the information you have provided. This error: > >> puppetdb.resnet.bris.ac.uk:8081: Connection refused - connect(2) > > Its very rare that a bug in a running piece of code/framework whatever > will cause a connection refused (destination port unreachable) message > on its own, its usually because the port and IP you are connecting to > is wrong and your client never got to connect to your application. > Thus its the kernel that returns the error, not the application. > > So generally, this doesn't marry up in my mind with this error message: > >> 2013-11-27 12:09:58,347 WARN [qtp1710594959-45] [io.nio] >> javax.net.ssl.SSLException: Inbound closed before receiving peer's >> close_notify: possible truncation attack? > > This implies you _did_ connect. In short I almost believe these are > somehow unrelated, or we're mixing errors here. The SSL error is most > definitely concerning, but doesn't make sense with the connection > refused message. A connection refused usually happens long before the > client gets to the serving application, if you see what I mean :-). > > Can you test the port with 'telnet puppetdb.resnet.bris.ac.uk 8081' > from the puppet master and confirm the connection refused manually? > Also - can you make sure these errors truly to correlate? Try to > reproduce both at the same time if you can. Also make sure no other > traffic is going to the PuppetDB web server at the same time. > > The details for how the master connects to the PuppetDB instance is in > /etc/puppet/puppetdb.conf, double check these are correct and that the > hostname resolves to what you think it does. Also check you don't have > any firewalling enabled, its rare but firewalls can throw destination > port unreachable also. > > Now the SSL error is valid and concerning to me on a separate level. I > have a whole bunch of questions though: > > * What _exact_ version of the JDK is PuppetDB using? The output of > 'jinfo <pid>' (pid of the jvm process for puppetdb) would be helpful > here, and the exact package revision from Centos. > * What exact version of PuppetDB are you running? > * Are you sure it was just openssl that was upgraded? Not java as > well? Double check your yum.log or whatever. > * Have you tried downgrading the recently upgraded packages to see if > it solves it? If it was an upgrade that caused it, a downgrade and > restart of PuppetDB should solve it in theory. I'd be interested if > this works, and what packages you downgraded to. > * Can you show the full stack trace from the PuppetDB log, if there is > more to it. >
If your runnning jdk 6u26 or older you're probably hitting these bugs. I had this same error with the OpenDJ LDAP server a few years back and upgrading the JDK fixed it. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6932403 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7025227 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CADaviKt5DNnOGb31_-1dWeXpEGX7L3eG34RBx5%3DYPGcCwXgDZQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
