----- Original Message ----- > From: "Martin Willemsma" <mwillem...@gmail.com> > To: puppet-users@googlegroups.com > Sent: Friday, June 8, 2012 7:11:39 AM > Subject: Re: [Puppet Users] MCollective not all nodes answer to commands when > using aes_security plugin > > Thanks for your response. > One thing I noticed when using the PSK is that I indeed see the > discovery with the progressbar. That's one thing I never see when > using AES. Commands always come back from discovered nodes when using > PSK. > > You suggest SSL TLS, is that the same as AES provider i'm using right > now?
identity of the client is securely established and the payload is encrypted using industry standards, i guess it depends on your needs though > > I run the client and on the same node as the rabbitmq. Also tried > with an activemq installation on another node in the same subnet. > Doesn't seem to make any difference. The node running the messagebus > is mostly idle. 4 CPUs / 4Gb ram and on the other node 2 CPU / 2 Gb > ram. I also tried this the client on my workstation. I5 / 8 Gb ram / > SSD disk, same behavior. > > I agree that the ping times are pretty high but I could live with > that if at least all the replies came back. ping times that long will just prevent everything from working. there's a fundamental problem somewhere. > I spend quite some time making this work on our platform. I need to > look more in-depth on the network part. > > 2012/6/7 R.I.Pienaar <r...@devco.net>: > > > > > > ----- Original Message ----- > >> From: "Martin Willemsma" <mwillem...@gmail.com> > >> To: "Puppet Users" <puppet-users@googlegroups.com> > >> Sent: Thursday, June 7, 2012 7:11:41 AM > >> Subject: [Puppet Users] MCollective not all nodes answer to > >> commands when using aes_security plugin > >> > >> Hi, > >> > >> I deployed MCollective to our Puppet clients. approx. ~ 200. Our > >> platform requires the most secure setup possible, so PSK as > >> securityprovider is not an option. > > > > I'd almost always suggest SSL TLS + the ssl plugin now. > > > >> Therefor I changed the security provider to aes_security reusing > >> puppets certificates in the server.cfg as found in the docs (1) > >> Our goal is to use mcollective to offload event-driven actions to > >> agents running on designated nodes from a webapplication. > >> > >> e.g: send out a message to the 'platform' collective to create a > >> DNS > >> record. This message should be processed by a node that runs the > >> 'DNS' agent. > >> > >> One thing I noticed after switching to the aes_security plugin is > >> the > >> ping latency went up and a reply to an action does not come back > >> from > >> all the nodes. Were does this latency come from? > >> If I do a mco ping on the client I expect: > >> > >> - every node to respond > >> - show me the ---- ping statistics ---- in the end > >> - jump back to my console ready for the next command > >> > >> but it does not. Instead it shows me the output for 207 nodes and > >> then > >> it just "HANGS" there. > > > >> > >> This output shows pingtimes hostnames omitted > >> > >> 1340.38 ms <- first reply > >> 1406.25 ms > >> 1456.71 ms > >> 1508.19 ms > >> 1550.52 ms > >> 1576.07 ms > >> 1601.15 ms > >> 1627.40 ms > >> 1653.23 ms > >> 1678.26 ms > >> [ .. omitted intentionally ] > >> 7518.66 ms > >> 7556.47 ms > >> 7593.06 ms > >> 7623.46 ms > >> 7648.64 ms > >> 7685.62 ms > >> 7722.84 ms <- last reply I see on the client console > > > > > > There are a few odd things here, the first reply is way too slow, > > the > > AES plugin is computationally very heavy and not suited for large > > deploys yours though is not large and even then the overhead is in > > the > > 30 to 40ms over that of the PSK plugin on first response - the > > effect > > snow balls but this should not be the performance I expect. > > > > Second the 'mco ping' should not run indefinitely till you stop it, > > it > > should run for 5 seconds and then end, does yours do that with the > > PSK > > plugin active? > > > > Hard to guess what might be the underlying cause for the above > > combination of issues - could be a very slow machine as the mco > > client, > > could be issues on the network perhaps there are a lot of TCP > > rebroadcasts or something along those lines. > > > > On the machines that do not respond do you see anything in their > > logs - > > put them in debug and make sure they got the request and replied. > > Anything weird on your broker? Large CPU usage perhaps? > > > > > >> > >> > >> If I check the the logfile on the client sending the command > >> '/var/log/mcollective.log' the last few lines show me: > >> > >> D, [2012-06-07T07:39:46.470905 #15910] DEBUG -- : > >> pluginmanager.rb:83:in `[]' Returning cached plugin > >> security_plugin > >> with class MCollective::Security::Aes_security > >> D, [2012-06-07T07:39:46.471029 #15910] DEBUG -- : > >> aes_security.rb:202:in `deserialize' De-Serializing using marshal > >> D, [2012-06-07T07:39:46.471121 #15910] DEBUG -- : > >> aes_security.rb:255:in `decrypt' Decrypting message using private > >> key > >> D, [2012-06-07T07:39:46.495265 #15910] DEBUG -- : > >> aes_security.rb:202:in `deserialize' De-Serializing using marshal > >> D, [2012-06-07T07:39:46.495711 #15910] DEBUG -- : stomp.rb:191:in > >> `receive' Waiting for a message from Stomp > >> > >> I can wait forever but it does not receive > >> I use (control + break) to exit out > >> > >> ^C > >> > >> ---- ping statistics ---- > >> 207 replies max: 6877.20 min: 616.98 avg: 3912.99 > >> > >> Logfile shows me: > >> > >> D, [2012-06-07T07:41:10.571316 #15910] DEBUG -- : client.rb:72:in > >> `unsubscribe' Unsubscribing reply target for discovery > >> D, [2012-06-07T07:41:10.571496 #15910] DEBUG -- : > >> pluginmanager.rb:83:in `[]' Returning cached plugin > >> connector_plugin > >> with class MCollective::Connector::Stomp > >> D, [2012-06-07T07:41:10.571615 #15910] DEBUG -- : stomp.rb:257:in > >> `unsubscribe' Unsubscribing from > >> /topic/mcollective.discovery.reply > >> D, [2012-06-07T07:41:10.572767 #15910] DEBUG -- : > >> pluginmanager.rb:83:in `[]' Returning cached plugin > >> connector_plugin > >> with class MCollective::Connector::Stomp > >> D, [2012-06-07T07:41:10.572849 #15910] DEBUG -- : stomp.rb:264:in > >> `disconnect' Disconnecting from Stomp > >> > >> Same behavior with using any of the other commands 'get_fact' , > >> 'rpc > >> package' 'rpc service'. I'm just not able to do a search over the > >> collective when using the AES plugin. > >> > >> If I switch switch back to PSK replies are speedy and always come > >> back. But then again this is not want. > >> > >> At first I was using RabbitMQ default config. I tries some > >> tweaking > >> but did not seem to make any difference to the behaviour of mco. I > >> switched to ActiveMQ 5.6 with the configfiles from puppetlabs.git. > >> Set > >> it up according to the docs , again played with some setttings and > >> did > >> not do anything at all. > >> > >> tcpdumps show the node running the mcollective server responds to > >> the > >> message send from the mcollective client. But seconds after the > >> node > >> replies the output gets printed on the client. Somehow it looks > >> like > >> the message gets 'STUCK' in the messagebus and arrives late on the > >> client. > >> > >> Any hints on were to tackle this issue are more then welcome and > >> really appreciated . This issue is blocking the implementation of > >> mcollective on our platform which is more than just sad > >> > >> Currently I'm using MCollective 2.0.0 on Ubuntu 10.04 LTS X86_64. > >> > >> (1) > >> http://docs.puppetlabs.com/mcollective/reference/plugins/security_aes.html > >> > >> --- > >> Best regards, > >> > >> Martin > >> > >> -- > >> You received this message because you are subscribed to the Google > >> Groups "Puppet Users" group. > >> To post to this group, send email to > >> puppet-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >> puppet-users+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/puppet-users?hl=en. > >> > >> > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. > > > > > > -- > --- > Met vriendelijke groet, > > Martin Willemsma > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
