On 09/05/14 20:33, jcbollinger wrote:


On Thursday, May 8, 2014 8:58:52 AM UTC-5, Dirk Heinrichs wrote:

    Am 08.05.2014 15:34, schrieb Jonathan Gazeley:

    I suppose it would be possible to disable the agent from running
    as a daemon and use cron, and the cron job could easily use a
    MySQL handle as a locking device. But it still doesn't stop me
    from simply sshing to each of the nodes and forcing a puppet run,
    and breaking the cluster.

    Has anyone done anything like this before? Hope to have some
interesting ideas from you all :)

    You could

     1. enable splay on the client node


No, that would at best be unhelpful. Splay well tend to spread out the client load seen by the master over time, but it contributes nothing to avoiding runs on specific clients coinciding. In fact, it might increase the likelihood of /specific/ pairs (or triples) of clients' runs coinciding.

Yes, I read about the splay option and it seems like it wouldn't help in this case.

     1. use mcollective to orchestrate the agent runs. For example:
        "Update config file on all MariaDB servers, but only one at a
        time."


That, on the other hand, could be just the ticket, provided that it is acceptable to run the agent only via that mechanism (which itself could perhaps be triggered via cron to give automation). Even that, however, would not actively prevent client runs from coinciding if someone manually ran the agent on one of the sensitive systems.

We are in the very early stages of looking at what MCollective can do for us. It looks promising but at the time of writing we have basically zero experience with it :)


Consider this, however: puppet already employs a lock file to prevent multiple catalog runs from overlapping on the same system. What Jonathan asks is simply an extension of that mechanism. It could be achieved relatively easily if the systems in question shared the same lock file, and it turns out that the lock file name and location are configurable <http://docs.puppetlabs.com/references/latest/configuration.html#agentcatalogrunlockfile>. If the configured location where on a shared filesystem accessible to all the machines involved then I think the requested behavior would fall out pretty naturally.

This crossed my mind. NFS-mounted lock files seem like a disaster waiting to happen, though. I suppose the puppetmaster could host the NFS share and then if the puppetmaster or the network is down, the node wouldn't have been able to check in anyway.

I was also wondering if PuppetDB could be used for this way, or even a general purpose database with lock rows.

Note, however, that nothing is foolproof. A sufficiently authorized person could still override the lock file (simply by deleting it, for example) to allow multiple catalog runs to proceed simultaneously. At some point you just have to decide that your safeguards are good enough.

Indeed. There are only two of us that work full-time on puppet infrastructure and neither of us would delete a lock file without good reason. It just has to prevent accidentally starting a run by mistake, or cron/puppetd starting a run automatically. There's a lot to be said for "good enough" in the world of IT operations.

Cheers,
Jonathan



John

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com <mailto:puppet-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/3f93cc53-c867-4976-b071-b7ce5838417c%40googlegroups.com <https://groups.google.com/d/msgid/puppet-users/3f93cc53-c867-4976-b071-b7ce5838417c%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/53708C9D.3090404%40bristol.ac.uk.
For more options, visit https://groups.google.com/d/optout.

Reply via email to