Hi Henrik, I know we have some users who just batch all package installs up front. It'd be interesting to see if that was a feasible solution. it would by pass the graph entirely, which I'm sure could have problems, but it would, at least, be easy to build and understand. Would that suffice for a sufficient number of cases?
On Sep 13, 2013, at 9:52 AM, Henrik Lindberg <[email protected]> wrote: > Hi, > Ideas regarding a potential performance boost that can be gained by > performing batch processing of package installs/operations has been floating > around in the Puppet echo system for quite some time. > > There is a discussion (and a somewhat dated implementation/proposal) in > http://projects.puppetlabs.com/issues/2198 which is good background reading > for this topic. > > In issue #2198 (if you skipped reading it ;-)), the idea is that Puppet > should have the feature to install a list of packages given by the user. > > It seems doable to generalize this idea and let puppet automatically optimize > package installs under certain conditions. Performing individual package > installs is quite expensive and even if the optimization opportunities may > not be extensive (e.g. say that 20% (number completely made up) of packages > could at least be paired with one other package) this is still a worth while > activity. > > To kick this off, we need to do some research and design. So, here is an > attempt to get this started by asking a bunch of questions. > > Under what conditions can two (or more) packages operations be batched? > ----------------------------------------------------------------------- > As an example, say that a class contains a series of package resources > without any explicit dependencies between them. The idea is that this could > be optimized. Are there any conditions that makes this impossible? > > What if the resources are chained with explicit dependencies? (Guess is that > the dependencies were added for a reason, and should be done as individual > dependencies). > > What if the list of packages are of different type? Is an chain of implicit > dependencies between packages of the same type required to make it possible > to batch them? (Does it depend on the policy for implicit dependencies; > parse-order, random, etc.)? > > What if there are other implicit dependencies. Can it be deduced that an > intermixed resource has no effect on the outcome of a following package > operation? (Exec's can for sure do things). > > Is it possible to optimize across class boundaries? > > Is it enough to look at the queue of actions in the "planned catalog" and > simply look-ahead for packages handled by the same provider. An unbroken > chain of operations handled by the same provider is collected and then handed > off to the provider? Does this provide enough optimization, or are we then > likely to miss optimization opportunities? > > How can we collect metrics for this? > > What needs to be done to providers? > ----------------------------------- > Clearly the capability to handle multiple requests must be implemented for > package managers that support this. What should the API look like? > > What needs to be done to the Package type? > ------------------------------------------ > Is it all an ordering issue and handing off resources to the provider, or do > we need to do things to the Package type as well? > > Are there situations were it is of value to veto batching per resource? > (depending on how much optimization than can be deduced by looking at > resource-dependencies). > > Explicit group/list? > -------------------- > If we want users to be able to explicitly give a group of packages to manage > - how should that work? A new resource type? An attribute on Package? A > defined type? > > If we cannot optimize across classes, can we support explicit grouping/batch > operation? (Seems complex with yet another containment hierarchy - or can > this be done by introspecting a dependency chain of custom resources/classes > perhaps used specifically for this purpose? > > - henrik > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev. > For more options, visit https://groups.google.com/groups/opt_out. -- Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
