On Tue, Aug 27, 2013 at 1:02 PM, Erik Dalén <[email protected]>wrote:
> ruby ./load_the_world.rb --parser future > user system total real > require 0.400000 0.060000 0.460000 ( 0.486126) > init 0.010000 0.000000 0.010000 ( 0.010955) > import 38.580000 0.280000 38.860000 ( 38.970678) > Stats > Known resource classes: 1034 > Known resource defines: 113 > Known resource nodes: 0 > Manifests loaded: 1141 > > If this is going to add 39s to every compile it won't be very good for us. > (this is not exactly the same hardware as most puppetmasters we have > though, but on the other hand they do many simultaneous compiles). > > Yeah, that wouldn't be good. I ran --parser future locally on my little set of testing data and found that --parser future is 20-30x slower than the current parser. Dalen, could you try the same thing, but without --parser future? I'd be interested if you see the same difference. Also, I don't think this would add 39s to every compile on the puppetmaster. It would only do that if it needs to reload. Based on earlier discussion, I think this would be best done from a signal. Even then there are techniques we could use to offload that (pre-parse and load serialized AST, for instance). This time would, currently, likely show up in puppet apply runs. I don't think it would be a good idea to cause puppet apply for testing purposes to jump from 1 or 2 seconds to 39+. > On the other hand we are working on a system to create dynamic per node > environments that only include the modules that node needs, so then the > difference might not be that big compared to the current method. > > > > On 27 August 2013 17:48, Andy Parker <[email protected]> wrote: > >> On Mon, Aug 26, 2013 at 4:13 PM, Joshua Hoblitt <[email protected]>wrote: >> >>> On 08/26/2013 11:41 AM, Andy Parker wrote: >>> > How would you handle reloading in this scenario? >>> > >>> > >>> > I can think of two ways: track all of the files loaded and reload when >>> > something changes or is added, reload only when told. I lean towards >>> the >>> > "reload only when told" for a production master, since then the runtime >>> > behavior of it is much more predictable in time (no more stating) and >>> in >>> > catalogs produced (no more ending up with half of one manifest version >>> > and half of another). >>> >>> I think the "load everything" and "only reload when told to" approach is >>> the correct one for the master. I suspect it will solve the random >>> classes disappearing while pushing an update problem I've seen in the >>> wild (but can't reproduce on demand) and definitely the fall out I see >>> when trying to change the git branch for an entire environment. This is >>> a very logical workflow and one many folks are already accustomed to >>> from HUPing bind, etc. >>> >>> I'd advocate caution about making "greedy loading" the default for local >>> apply. Syntastic can already be painful slow. :( >>> >>> >> Yeah, I think if it turns out that greedy loading is prohibitively >> expensive our best bet is to a) make the parser faster, b) expose parts of >> the code better so that specific use cases can be done more efficiently, >> and only after that c) put laziness back in. >> >> In order to answer the question of how long import everything would take, >> I've put together a simple script that I think anyone can run to get some >> information. It would be great if some people with large sets of manifests >> could try running this and see what kinds of numbers we get. For example >> this is what I get >> >> [10:46:17][Ruby(ruby-1.9.3-p392)][Git(master)] andy:puppet >> > be ./load_the_world.rb --parser future >> user system total real >> require 0.530000 0.100000 0.630000 ( 0.634259) >> init 0.000000 0.000000 0.000000 ( 0.005957) >> import 0.310000 0.020000 0.330000 ( 0.326274) >> Stats >> Known resource classes: 4 >> Known resource defines: 1 >> Known resource nodes: 0 >> Manifests loaded: 6 >> Manifests: >> /Users/andy/.puppet/manifests/site.pp >> /Users/andy/.puppet/modules/manifest1/manifests/foo.pp >> /Users/andy/.puppet/modules/manifest1/manifests/fool.pp >> /Users/andy/.puppet/modules/manifest1/manifests/init.pp >> /Users/andy/.puppet/modules/manifest2/manifests/init.pp >> /Users/andy/.puppet/modules/myfile/manifests/init.pp >> >> You can also pass it any parameters (such as environment or --parser >> future) >> >> Please run it, do any scrubbing of sensitive data you want and send back >> the results. >> >> >>> -Josh >>> >>> -- >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/puppet-dev. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Andrew Parker >> [email protected] >> Freenode: zaphod42 >> Twitter: @aparker42 >> Software Developer >> >> *Join us at PuppetConf 2014, September 23-24 in San Francisco* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-dev. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Erik Dalén > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev. > For more options, visit https://groups.google.com/groups/opt_out. > -- Andrew Parker [email protected] Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at PuppetConf 2014, September 23-24 in San Francisco* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
