Chris, To better help you, it would be great to know a few more things about your installation. First question: are you running puppetserver 5.0.0 or something later in the 5.x series (and is it the same on all servers)? Second, what version of the puppet-agent are on those servers? puppetserver 5.1.3 included a fix for https://tickets.puppetlabs.com/browse/SERVER-1922 which should improve performance some. To dig into what may be causing the compiles to be slower, I would recommend first checking out the client metrics. https://puppet.com/docs/puppetserver/5.1/http_client_metrics.html has some details, and I would be interested in the client metrics that page lists under the /puppet/v3/catalog. They are PuppetDB related requests, and as that was also upgraded alongside puppetserver it would be good to eliminate PuppetDB as a contributor. PuppetDB slowness can show up as slow catalog compiles, which in turn will hold jrubies for longer and might explain some of what you are seeing.
On Mon, Jan 8, 2018 at 7:31 PM, chris smith <dmag...@gmail.com> wrote: > Hi there, > > I recently did an upgrade from puppetserver 2.7.2 to puppetserver 5.0 and > performance has bottomed out pretty terribly. Agents and puppetdb also got > updated. Compiling the catalog on the server used to take 10-20 seconds, now > they are taking 90-150 seconds and agent runs are taking 30+ minutes (used > to be a couple of minutes). > > The architecture is distributed, with: > * a central ca, running puppetserver, puppetdb, postgres 9.6 > * separate puppetservers replicated in other datacentres. These are also > running puppetdb, pointing writes to the central ca, but reads go to a > locally replicated database > > Other servers (agents) point to the replicated puppetservers to do all of > the work. > > The puppetservers were tuned (upped jvm memory, set max-instances). > > The architecture hasn't changed since the upgrade. > > The puppetservers are still running hiera3 configs, they haven't been > converted yet (it's on the list, but from what I read it wasn't a > showstopper). We have a reasonable amount of encrypted yaml files (using > hiera-eyaml-gpg), though this was the same as pre-upgrade and hasn't changed > significantly. > > Since the upgrade, I've tried re-tuning the jvm settings, changing > max-instances and not having much luck at all. I found the experimental > dashboard on the puppetservers and they show that there are no free jruby's, > but there has to be something I'm missing that's causing that to happen. > > I'm lost on what to look at next, is there an easy way to peak inside jruby > to see what's taking so long? > > Any suggestions would be great. > > Cheers, > Chris. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/4d0dc37f-c07e-4f8c-8323-44a90d68b208%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACD%3DwAf5eC%2B8mixcyYjUGAm-0jeBzYnq%2BOydg3ZNkJ5YWoROmg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
