I was able to get a bit of info about what the read queue is as the catalog is being executed. It looks like there are some times (this is just by eyeballing it, haven't crunched any numbers) where we can get a run of several packages that could be batched together.
The branch that contains the change for the extra logging is https://github.com/zaphod42/puppet/tree/spike/master/introspect-ready-queue I attached the output to the bug (http://projects.puppetlabs.com/issues/2198) since it is about 2MB. On Mon, Sep 16, 2013 at 9:12 AM, Andy Parker <[email protected]> wrote: > On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger <[email protected] > > wrote: > >> >> >> On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: >>> >>> >>> The problem with this picture for being able to batch operations >>> together, is that everything turns into calls on the Puppet::Type instance, >>> which then makes all of the individual calls to the provider. To batch, we >>> need to group resources together and then give the groups to the provider. >>> >> >> >> So you don't think my suggestion above to let providers assemble and >> apply batches is workable? I think it requires only one or two extra >> signals to the provider (via the Type), to mark batch boundaries. Most of >> the magic would happen in those providers that choose to perform it, and >> those that don't choose to perform it can just ignore the new signals. The >> main part of the protocol between the agent core and types/providers >> remains unchanged. >> >> I haven't delved into the details of how exactly it would be implemented, >> so perhaps there is a show-stopper there, but I'm not seeing a flaw in the >> basic idea. >> >> > No, you are right. I forgot about that one. I was just running through the > code, the biggest problem that I can see so far is simply that there isn't > "the provider". We end up with a provider instance per resource, as far as > I can tell. Others have solved that by tracking data on the provider class. > > I think that for the batching we just need a way of asking a provider if > two resources (for the same provider class) are batchable. The comparison > of batchable needs to be transitive (so if A and B are batchable, and so > are B and C, then all A, B, and C are). In fact it needs to also be > symmetric and reflexive, since it is really just another form of equality. > That helps us to define what can be batched together. > > Once we know that, then the problem is how to decide what exactly the > batches are. Since we don't actually have a complete view of all of the > resources, the decision is going to be based off of incomplete information. > Also the batches might need to take into account other factors that are out > of the control of the provider such as constraints from the "ordering" > selected. > > So, for instance, for "--ordering manifest" we would probably want to do a > kind of run length encoding of the resources. Start a batch on a resource, > and end the batch when the next resource is not batchable. > > >> >> John >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-dev. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Andrew Parker > [email protected] > Freenode: zaphod42 > Twitter: @aparker42 > Software Developer > > *Join us at PuppetConf 2014, September 23-24 in San Francisco* > -- Andrew Parker [email protected] Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at PuppetConf 2014, September 23-24 in San Francisco* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
