----- Original Message ----- > From: "Martin Willemsma" <mwillem...@gmail.com> > To: "Puppet Users" <puppet-users@googlegroups.com> > Sent: Thursday, June 7, 2012 7:11:41 AM > Subject: [Puppet Users] MCollective not all nodes answer to commands when > using aes_security plugin > > Hi, > > I deployed MCollective to our Puppet clients. approx. ~ 200. Our > platform requires the most secure setup possible, so PSK as > securityprovider is not an option.
I'd almost always suggest SSL TLS + the ssl plugin now. > Therefor I changed the security provider to aes_security reusing > puppets certificates in the server.cfg as found in the docs (1) > Our goal is to use mcollective to offload event-driven actions to > agents running on designated nodes from a webapplication. > > e.g: send out a message to the 'platform' collective to create a DNS > record. This message should be processed by a node that runs the > 'DNS' agent. > > One thing I noticed after switching to the aes_security plugin is the > ping latency went up and a reply to an action does not come back from > all the nodes. Were does this latency come from? > If I do a mco ping on the client I expect: > > - every node to respond > - show me the ---- ping statistics ---- in the end > - jump back to my console ready for the next command > > but it does not. Instead it shows me the output for 207 nodes and > then > it just "HANGS" there. > > This output shows pingtimes hostnames omitted > > 1340.38 ms <- first reply > 1406.25 ms > 1456.71 ms > 1508.19 ms > 1550.52 ms > 1576.07 ms > 1601.15 ms > 1627.40 ms > 1653.23 ms > 1678.26 ms > [ .. omitted intentionally ] > 7518.66 ms > 7556.47 ms > 7593.06 ms > 7623.46 ms > 7648.64 ms > 7685.62 ms > 7722.84 ms <- last reply I see on the client console There are a few odd things here, the first reply is way too slow, the AES plugin is computationally very heavy and not suited for large deploys yours though is not large and even then the overhead is in the 30 to 40ms over that of the PSK plugin on first response - the effect snow balls but this should not be the performance I expect. Second the 'mco ping' should not run indefinitely till you stop it, it should run for 5 seconds and then end, does yours do that with the PSK plugin active? Hard to guess what might be the underlying cause for the above combination of issues - could be a very slow machine as the mco client, could be issues on the network perhaps there are a lot of TCP rebroadcasts or something along those lines. On the machines that do not respond do you see anything in their logs - put them in debug and make sure they got the request and replied. Anything weird on your broker? Large CPU usage perhaps? > > > If I check the the logfile on the client sending the command > '/var/log/mcollective.log' the last few lines show me: > > D, [2012-06-07T07:39:46.470905 #15910] DEBUG -- : > pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin > with class MCollective::Security::Aes_security > D, [2012-06-07T07:39:46.471029 #15910] DEBUG -- : > aes_security.rb:202:in `deserialize' De-Serializing using marshal > D, [2012-06-07T07:39:46.471121 #15910] DEBUG -- : > aes_security.rb:255:in `decrypt' Decrypting message using private key > D, [2012-06-07T07:39:46.495265 #15910] DEBUG -- : > aes_security.rb:202:in `deserialize' De-Serializing using marshal > D, [2012-06-07T07:39:46.495711 #15910] DEBUG -- : stomp.rb:191:in > `receive' Waiting for a message from Stomp > > I can wait forever but it does not receive > I use (control + break) to exit out > > ^C > > ---- ping statistics ---- > 207 replies max: 6877.20 min: 616.98 avg: 3912.99 > > Logfile shows me: > > D, [2012-06-07T07:41:10.571316 #15910] DEBUG -- : client.rb:72:in > `unsubscribe' Unsubscribing reply target for discovery > D, [2012-06-07T07:41:10.571496 #15910] DEBUG -- : > pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin > with class MCollective::Connector::Stomp > D, [2012-06-07T07:41:10.571615 #15910] DEBUG -- : stomp.rb:257:in > `unsubscribe' Unsubscribing from /topic/mcollective.discovery.reply > D, [2012-06-07T07:41:10.572767 #15910] DEBUG -- : > pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin > with class MCollective::Connector::Stomp > D, [2012-06-07T07:41:10.572849 #15910] DEBUG -- : stomp.rb:264:in > `disconnect' Disconnecting from Stomp > > Same behavior with using any of the other commands 'get_fact' , 'rpc > package' 'rpc service'. I'm just not able to do a search over the > collective when using the AES plugin. > > If I switch switch back to PSK replies are speedy and always come > back. But then again this is not want. > > At first I was using RabbitMQ default config. I tries some tweaking > but did not seem to make any difference to the behaviour of mco. I > switched to ActiveMQ 5.6 with the configfiles from puppetlabs.git. > Set > it up according to the docs , again played with some setttings and > did > not do anything at all. > > tcpdumps show the node running the mcollective server responds to the > message send from the mcollective client. But seconds after the node > replies the output gets printed on the client. Somehow it looks like > the message gets 'STUCK' in the messagebus and arrives late on the > client. > > Any hints on were to tackle this issue are more then welcome and > really appreciated . This issue is blocking the implementation of > mcollective on our platform which is more than just sad > > Currently I'm using MCollective 2.0.0 on Ubuntu 10.04 LTS X86_64. > > (1) > http://docs.puppetlabs.com/mcollective/reference/plugins/security_aes.html > > --- > Best regards, > > Martin > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
