On Wednesday, March 16, 2016 at 4:36:09 AM UTC-5, Jelle Smet wrote: > > > On Tuesday, March 15, 2016 at 3:50:28 PM UTC+1, jcbollinger wrote: >> >> >> >> On Monday, March 14, 2016 at 11:23:53 AM UTC-5, Jelle Smet wrote: >>> >>> Hi list, >>> >>> >>> I have a Puppet module (internal to the company) which makes use of >>> exported resources. >>> The exported resource data stored in PuppetDB is used to configure an >>> external application. >>> >> >> >> That sounds like an odd way to go about it. One would normally export >> resources that directly configure the external application, one way or >> another. Details vary greatly with nature of the application that needs to >> be configured, of course. >> > > > So we have an internal restful service which has to be configured by > submitting JSON data into the appropriate api endpoints. > The JSON data can be constructed from data stored in PuppetDB. >
As I said, that's a very odd way of doing things. It's odd on at least two levels: 1. it's a bit odd that you want to configure the application by dynamically submitting data to the service, instead of by directly managing its configuration data store. This approach has negative consequences, including, but not limited to, the type of problem you have asked about. 2. it's very odd that you construct the JSON data by using an external application to cull it from PuppetDB, especially when you're already using exported resources. There may be a good reason for this approach -- it's hard to tell since you're so parsimonious with the details -- but I wouldn't generally recommend it. Overall, it does not mesh well with Puppet's approach to configuration management: state, not actions, are the central focus of Puppet's view of configuration management. Its basic mode of operation involves cycles of (a) determining what the desired machine state is, (b) comparing the actual state to the desired one, and (c) updating the machine state where necessary. The process you describe seems grossly inconsistent with that model. > We have a process which reads the content of puppetdb, constructs the JSON > and submits that to the necessary api end points of the restful webservice. > The problem with this approach is that the middleware process (responsible > for reading puppetdb, converting & submitting the data to the api endpoint) > continuously needs to read puppetdb for added and removed hosts. > This is simply not a very practical thing to do and causes a lot IO whilst > nothing needs to be done. > I agree, continuously polling the DB is a terrible idea. > > The idea is to trigger this middleware to read from PuppetDB if the > exported resources of a host have changed. Because only then it needs to > update the configuration in the restful interface. > That would be better because you don't need to continuously scrape > puppetdb for changes. > > A better idea would be to cut out the middleware altogether. Put the the host on which your service runs under Puppet management, if it isn't already. Manage the service's configuration files via Puppet, and signal it to re-read them when Puppet changes them. This is a pretty standard pattern. A variation on this pattern that might interest you involves building the configuration file from multiple fragments, via the Concat module. The fragments can be exported by multiple hosts and collected into the catalog for the machine(s) on which the overall file needs to be managed. Even if there is no alternative to managing your service via its REST API -- maybe it has no static configuration file, for example -- you should still be able to use this or a similar approach to avoid scraping the needed configuration data from PuppetDB. > If puppet (agent/master/whatever puppet component) could produce a log > event (like syslog) when the exported resources of a host have changed that > would be great. > If you *must* do it this way, then you can configure agents to send reports to the master at the end of each run, and you can configure a report processor on the master that does whatever it needs to do with the resulting data. If you can get by with less data, and if your catalog runs normally go complete without any changes, then you could perhaps also run the agent via a scheduler (instead of as a daemon), with the --detailed-exit-codes option turned on, and trigger your middleware based on the agent reporting via its exit code that changes were applied. > > The constraints I got: > > - No execs because it's considered to be bad practice. > > Absurd. You should use Execs where they make sense. Execs can be over-used, but the alternative is to use a different resource type, not to avoid modeling your configuration via resources. If there isn't already a suitable resource type available, then that involves making one. And don't tell whoever enforces that policy, but you'll see Execs in a lot of popular and well-respected modules, including many of those published by PuppetLabs. If you rely on many third-party modules then you probably have some Execs in your manifest set from those sources. If the prohibition against Execs is absolute and incontrovertible, however, then it wouldn't be too hard to write a custom type that just wraps the specific command you need to run, and that executes it (only) when it refreshes. Deploying an instance of such a type in the right place would be a great deal better than relying on some kind of external middleware to trigger updates based on log traffic. > > - Adding triggers to Postgress is not allowed because it requires > modifications to the DB schema and might impact upgrade procedures to > deviate from trom what is provided by Puppetlabs > > That's reasonable. I briefly considered suggesting triggers, but rejected the idea for exactly the reason you give. > > - The Puppet is not allowed to contact / have a dependency on any > external networked services (local syslog would be fine though). hence no > puppet-tell > > Ok. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/fc737f55-0aaa-4645-ad15-97458812e1ea%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
