On Wed, Oct 6, 2010 at 10:43 AM, Nigel Kersten <ni...@explanatorygap.net>wrote:
> On Tue, Oct 5, 2010 at 7:39 PM, Jeff McCune <j...@puppetlabs.com> wrote: > > On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi <lrh...@gmail.com> wrote: > >> Hello all, > >> > >> This is not a puppet proper issue of course.. but I was wondering if > >> any of you could share some thoughts... > >> > >> When you deploy a system like Puppet at a large park of systems, you > >> instantly increase the efficacy of mistakes and bugs in destroying the > >> environment.... > > > > With great power comes great responsibility. > > > >> How do you deal with that? I would be interested in any experience or > >> input, especially puppet proper best practices... > > > > In addition to the other responses (testing, some more testing, and a > > well defined release process) I also recommend a "Big red stop button" > > of sorts. Few sites seem to employ this, but the ones who have have > > reported it to be incredibly useful when necessary and enjoy a little > > more peace of mind. > > > > The big red stop button can take many forms. I personally use a > > wrapper script to launch puppet agent runs from cron, so the stop > > button in my case could be a simple curl command before running > > puppet agent and the agent proceeds if and only if the curl command > > does not get a valid 200 response from the HTTP request. > > > > This allow you to quickly touch a file somewhere and know it will halt > > all puppet agents on all of your nodes. > > > We have a slightly more sophisticated BigRedButton that we use. > > The clients make an HTTP request, supplying some core facts as > parameters, like domain, hardware-type, major OS version, release > track, etc. > > If they get anything other than a 404 for the response, they stop. > > Server side it's something you could whip up in any language quickly, > where we have a simple YAML file that the app reads to work out > whether it should be returning non-404 depending upon the parameters > provided. > > This allows you to have slightly more fine-grained control over a > disparate fleet, where you can block individual geographic regions, > releases, hardware models from executing their normal update. > > You could implement all of this in puppet pre-run hooks nowadays > rather than a wrapper script, but we have other reasons to want a > wrapper. Admittedly these reasons are getting smaller and smaller each > puppet release. :) > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-us...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com<puppet-users%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > Write a script that does a diff between what's in revision control, and what's in /etc/puppet. Show those changes to the user and prompt before updating the local working copy. Then, do a puppetrun. Doug. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
