On 2019-02-11 14:42, Mike Sharpton wrote:
Hey all,
We have recently upgraded our environment from Puppetserver 4.2.2 to
Puppetserver 6.0.2. We are running a mix of Puppet 4 and Puppet 6
agents until we can get them all upgraded to 6. We have around 6000
nodes, and we had 4 Puppetservers, but we added two more due to capacity
issues with Puppet 6. The load is MUCH higher with Puppet 6. To the
question, I am seeing longer and longer agent run times after about two
days of the services running. The only error in the logs that seems to
have any relation to this is this string.
2019-02-11T04:32:28.409-06:00 ERROR [qtp1148783071-4075] [p.r.core]
Internal Server Error: java.io.IOException:
java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
After I restart the puppetserver service, this goes away for about two
days. I think Puppetserver is dying a slow death under this load (load
average of around 5-6). We are running Puppetserver on vm's that are
10X8GB and using 6 Jruby workers per Puppetserver and a 4GB heap. I
have not seen any OOM exceptions and the process never crashes. Has
anyone else seen anything like this? I did some Googling and didn't
find a ton of relevant stuff. Perhaps we need to upgrade to the latest
version to see if this helps? Even more capacity? Seems silly. Thanks
in advance!
There may be a slow memory leak that over time makes the server busy win
non productive work (scanning for garbage on an ever increasing heap).
If you were to increase capacity you would risk only changing the 2 days
to a couple more, but not actually solving the issue.
Try to look at server memory usage over the two days.
Also, naturally, upgrade to latest and make sure modules are updated as
well.
Do you by any chance have many environments with different versions of
ruby code? The environment isolation "puppet generate types" may be of
help if that is the case as loaded ruby resource types become sticky in
memory.
- henrik
Mike
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/q3shrm%241nb9%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.
