On 09/05/14 20:33, jcbollinger wrote:
On Thursday, May 8, 2014 8:58:52 AM UTC-5, Dirk Heinrichs wrote:
Am 08.05.2014 15:34, schrieb Jonathan Gazeley:
I suppose it would be possible to disable the agent from running
as a daemon and use cron, and the cron job could easily use a
MySQL handle as a locking device. But it still doesn't stop me
from simply sshing to each of the nodes and forcing a puppet run,
and breaking the cluster.
Has anyone done anything like this before? Hope to have some
interesting ideas from you all :)
You could
1. enable splay on the client node
No, that would at best be unhelpful. Splay well tend to spread out
the client load seen by the master over time, but it contributes
nothing to avoiding runs on specific clients coinciding. In fact, it
might increase the likelihood of /specific/ pairs (or triples) of
clients' runs coinciding.
Yes, I read about the splay option and it seems like it wouldn't help in
this case.
1. use mcollective to orchestrate the agent runs. For example:
"Update config file on all MariaDB servers, but only one at a
time."
That, on the other hand, could be just the ticket, provided that it is
acceptable to run the agent only via that mechanism (which itself
could perhaps be triggered via cron to give automation). Even that,
however, would not actively prevent client runs from coinciding if
someone manually ran the agent on one of the sensitive systems.
We are in the very early stages of looking at what MCollective can do
for us. It looks promising but at the time of writing we have basically
zero experience with it :)
Consider this, however: puppet already employs a lock file to prevent
multiple catalog runs from overlapping on the same system. What
Jonathan asks is simply an extension of that mechanism. It could be
achieved relatively easily if the systems in question shared the same
lock file, and it turns out that the lock file name and location are
configurable
<http://docs.puppetlabs.com/references/latest/configuration.html#agentcatalogrunlockfile>.
If the configured location where on a shared filesystem accessible to
all the machines involved then I think the requested behavior would
fall out pretty naturally.
This crossed my mind. NFS-mounted lock files seem like a disaster
waiting to happen, though. I suppose the puppetmaster could host the NFS
share and then if the puppetmaster or the network is down, the node
wouldn't have been able to check in anyway.
I was also wondering if PuppetDB could be used for this way, or even a
general purpose database with lock rows.
Note, however, that nothing is foolproof. A sufficiently authorized
person could still override the lock file (simply by deleting it, for
example) to allow multiple catalog runs to proceed simultaneously. At
some point you just have to decide that your safeguards are good enough.
Indeed. There are only two of us that work full-time on puppet
infrastructure and neither of us would delete a lock file without good
reason. It just has to prevent accidentally starting a run by mistake,
or cron/puppetd starting a run automatically. There's a lot to be said
for "good enough" in the world of IT operations.
Cheers,
Jonathan
John
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/3f93cc53-c867-4976-b071-b7ce5838417c%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-users/3f93cc53-c867-4976-b071-b7ce5838417c%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/53708C9D.3090404%40bristol.ac.uk.
For more options, visit https://groups.google.com/d/optout.
