Hello Henrik, The heap being at 4GB is all the higher I would raise it, as you say GC becomes costly with big heaps. The memory usage ramps up quite quickly to well above the configured max heap within minutes. It comes up to about 5.8GB of usage quickly as we manage many resources on many nodes. We do not have many environments. We normally have a production branch and only use a preprod branch to move changes up our environment with a module that managed the puppet.conf on our nodes. We will keep looking, but I don't see a smoking gun. Anyone else have any ideas? Puppet 4 was able to handle this load with only 4 JRuby workers and 4 Puppet servers. Thanks for your help,
Mike On Monday, February 11, 2019 at 2:06:11 PM UTC-6, Henrik Lindberg wrote: > > On 2019-02-11 14:42, Mike Sharpton wrote: > > Hey all, > > > > We have recently upgraded our environment from Puppetserver 4.2.2 to > > Puppetserver 6.0.2. We are running a mix of Puppet 4 and Puppet 6 > > agents until we can get them all upgraded to 6. We have around 6000 > > nodes, and we had 4 Puppetservers, but we added two more due to capacity > > issues with Puppet 6. The load is MUCH higher with Puppet 6. To the > > question, I am seeing longer and longer agent run times after about two > > days of the services running. The only error in the logs that seems to > > have any relation to this is this string. > > > > 2019-02-11T04:32:28.409-06:00 ERROR [qtp1148783071-4075] [p.r.core] > > Internal Server Error: java.io.IOException: > > java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 > ms > > > > > > After I restart the puppetserver service, this goes away for about two > > days. I think Puppetserver is dying a slow death under this load (load > > average of around 5-6). We are running Puppetserver on vm's that are > > 10X8GB and using 6 Jruby workers per Puppetserver and a 4GB heap. I > > have not seen any OOM exceptions and the process never crashes. Has > > anyone else seen anything like this? I did some Googling and didn't > > find a ton of relevant stuff. Perhaps we need to upgrade to the latest > > version to see if this helps? Even more capacity? Seems silly. Thanks > > in advance! > > > > There may be a slow memory leak that over time makes the server busy win > non productive work (scanning for garbage on an ever increasing heap). > If you were to increase capacity you would risk only changing the 2 days > to a couple more, but not actually solving the issue. > > Try to look at server memory usage over the two days. > > Also, naturally, upgrade to latest and make sure modules are updated as > well. > > Do you by any chance have many environments with different versions of > ruby code? The environment isolation "puppet generate types" may be of > help if that is the case as loaded ruby resource types become sticky in > memory. > > - henrik > > > Mike > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to puppet-users...@googlegroups.com <javascript:> > > <mailto:puppet-users+unsubscr...@googlegroups.com <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com > > > < > https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > > -- > > Visit my Blog "Puppet on the Edge" > http://puppet-on-the-edge.blogspot.se/ > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/59932f88-0416-45d3-b938-0f3c417051fa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
