On Tue, Oct 5, 2010 at 7:39 PM, Jeff McCune <j...@puppetlabs.com> wrote: > On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi <lrh...@gmail.com> wrote: >> Hello all, >> >> This is not a puppet proper issue of course.. but I was wondering if >> any of you could share some thoughts... >> >> When you deploy a system like Puppet at a large park of systems, you >> instantly increase the efficacy of mistakes and bugs in destroying the >> environment.... > > With great power comes great responsibility. > >> How do you deal with that? I would be interested in any experience or >> input, especially puppet proper best practices... > > In addition to the other responses (testing, some more testing, and a > well defined release process) I also recommend a "Big red stop button" > of sorts. Few sites seem to employ this, but the ones who have have > reported it to be incredibly useful when necessary and enjoy a little > more peace of mind. > > The big red stop button can take many forms. I personally use a > wrapper script to launch puppet agent runs from cron, so the stop > button in my case could be a simple curl command before running > puppet agent and the agent proceeds if and only if the curl command > does not get a valid 200 response from the HTTP request. > > This allow you to quickly touch a file somewhere and know it will halt > all puppet agents on all of your nodes.
We have a slightly more sophisticated BigRedButton that we use. The clients make an HTTP request, supplying some core facts as parameters, like domain, hardware-type, major OS version, release track, etc. If they get anything other than a 404 for the response, they stop. Server side it's something you could whip up in any language quickly, where we have a simple YAML file that the app reads to work out whether it should be returning non-404 depending upon the parameters provided. This allows you to have slightly more fine-grained control over a disparate fleet, where you can block individual geographic regions, releases, hardware models from executing their normal update. You could implement all of this in puppet pre-run hooks nowadays rather than a wrapper script, but we have other reasons to want a wrapper. Admittedly these reasons are getting smaller and smaller each puppet release. :) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
