There's some places where there is 19 packages on the ready queue in that example catalog.
On 17 September 2013 18:48, Andy Parker <[email protected]> wrote: > I was able to get a bit of info about what the read queue is as the > catalog is being executed. It looks like there are some times (this is just > by eyeballing it, haven't crunched any numbers) where we can get a run of > several packages that could be batched together. > > The branch that contains the change for the extra logging is > https://github.com/zaphod42/puppet/tree/spike/master/introspect-ready-queue > > I attached the output to the bug ( > http://projects.puppetlabs.com/issues/2198) since it is about 2MB. > > > On Mon, Sep 16, 2013 at 9:12 AM, Andy Parker <[email protected]> wrote: > >> On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger < >> [email protected]> wrote: >> >>> >>> >>> On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: >>>> >>>> >>>> The problem with this picture for being able to batch operations >>>> together, is that everything turns into calls on the Puppet::Type instance, >>>> which then makes all of the individual calls to the provider. To batch, we >>>> need to group resources together and then give the groups to the provider. >>>> >>> >>> >>> So you don't think my suggestion above to let providers assemble and >>> apply batches is workable? I think it requires only one or two extra >>> signals to the provider (via the Type), to mark batch boundaries. Most of >>> the magic would happen in those providers that choose to perform it, and >>> those that don't choose to perform it can just ignore the new signals. The >>> main part of the protocol between the agent core and types/providers >>> remains unchanged. >>> >>> I haven't delved into the details of how exactly it would be >>> implemented, so perhaps there is a show-stopper there, but I'm not seeing a >>> flaw in the basic idea. >>> >>> >> No, you are right. I forgot about that one. I was just running through >> the code, the biggest problem that I can see so far is simply that there >> isn't "the provider". We end up with a provider instance per resource, as >> far as I can tell. Others have solved that by tracking data on the provider >> class. >> >> I think that for the batching we just need a way of asking a provider if >> two resources (for the same provider class) are batchable. The comparison >> of batchable needs to be transitive (so if A and B are batchable, and so >> are B and C, then all A, B, and C are). In fact it needs to also be >> symmetric and reflexive, since it is really just another form of equality. >> That helps us to define what can be batched together. >> >> Once we know that, then the problem is how to decide what exactly the >> batches are. Since we don't actually have a complete view of all of the >> resources, the decision is going to be based off of incomplete information. >> Also the batches might need to take into account other factors that are out >> of the control of the provider such as constraints from the "ordering" >> selected. >> >> So, for instance, for "--ordering manifest" we would probably want to do >> a kind of run length encoding of the resources. Start a batch on a >> resource, and end the batch when the next resource is not batchable. >> >> >>> >>> John >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/puppet-dev. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Andrew Parker >> [email protected] >> Freenode: zaphod42 >> Twitter: @aparker42 >> Software Developer >> >> *Join us at PuppetConf 2014, September 23-24 in San Francisco* >> > > > > -- > Andrew Parker > [email protected] > Freenode: zaphod42 > Twitter: @aparker42 > Software Developer > > *Join us at PuppetConf 2014, September 23-24 in San Francisco* > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev. > For more options, visit https://groups.google.com/groups/opt_out. > -- Erik Dalén -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
