Thanks R.I. Pienaar, that's exactly the kind of suggestion I was hoping
for! :)
I haven't written any Ruby for ~9 years, but as I regularly develop in
several other languages (primarily C++), I'll enjoy getting my hands dirty
with Ruby again.
And thanks for including a basic single-interpreter example - that will
help me get going a lot quicker!
Thanks,
Paul.
On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:
>
> hello,
>
> ----- Original Message -----
> > From: "Paul Colby" <pco...@gmail.com <javascript:>>
> > To: puppet...@googlegroups.com <javascript:>
> > Sent: Monday, August 13, 2012 6:22:10 PM
> > Subject: [Puppet Users] hiera command line performance
> >
> > Hi everyone,
> >
> >
> > The hiera command line program (version 0.3.0) is currently taking
> > around 2 seconds per invocation. This seems quite slow to me, but is
> > probably not an issue for most users.
>
> 2 seconds seem quite slow though i guess it depends on the size of your
> yamls and so forth, sounds like yours are big and complex. I'd hope the
> bulk of the time is spent on starting/stopping ruby each time.
>
> A better option might be to consider writing your script in ruby rather
> than bash so you can then do your lookup that way reusing the single
> hiera instance and avoid the ruby start/stop cost
>
> So a simple test on my setup doing:
>
> for i in {1..100}
> do
> hiera syslocation ::location=foo
> done
>
> this takes 30 seconds, ages. Here is the same in Ruby reusing the
> class and using a single interpreter:
>
> h = Hiera.new(:config => "/etc/hiera.yaml")
>
> Benchmark.measure do
> 100.times do
> puts h.lookup("syslocation", "", {"::location" => "foo"})
> end
> end
>
> this takes 0.1 second for 100 lookups. If you know a bit of ruby this
> would be a good approach for you even if you just write your script to
> configure a single machine in ruby and run it 40 times only
>
> >
> >
> > We have a number of legacy machines that for one reason or another
> > have not been puppetised yet, and probably won't be anytime soon, so
> > I have a somewhat clever script that, for each of these hosts, uses
> > the hiera command like utility to export a few puppet-templated
> > documents just as puppet+hiera does for our puppeted server. This
> > works very nicely, but involves ~250 hiera lookups per host, which
> > at 2 seconds per lookup, is taking more than 8 minutes per host (and
> > using near 100% of a CPU core the entire time). As we have ~40 such
> > hosts, that's over 5 hours with no concurrency (we do use some
> > concurrency, but since each invocation uses near 100% of a CPU core
> > each, there's no point running more simultaneous hiera processes
> > than the number of CPUs, and even then, we need some reserved for
> > other services).
> >
> >
> > So, is there some way I can make this export run faster? The bulk of
> > the time seems to be spent in initialising the hiera command line
> > program, not the actual data lookup (for example, if I fail to
> > provide the necessary identity / scope files, it still takes 2
> > seconds to get around to returning an error, but if I fail to
> > provide a config file, it returns instantaneously with error). So,
> > I'd either like to make the program load / initialise much faster,
> > or be able to query multiple values for a single hiera command line
> > invocation.
> >
> >
> > Also, is there a more recent version of hiera that would be faster?
> > (I'm a little unclear about hiera versioning - we're running 0.3.0,
> > but I don't know if that's recent or old).
> >
> >
> > Any suggestions / tips would be greatly appreciated :)
> >
> >
> > Thanks.
> >
> >
> > Paul.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Puppet Users" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
> > To post to this group, send email to
> > puppet...@googlegroups.com<javascript:>.
>
> > To unsubscribe from this group, send email to
> > puppet-users...@googlegroups.com <javascript:>.
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
> >
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/WQV-NVyOHXgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
