On 2019-02-11 21:59, Mike Sharpton wrote:
Hello Henrik,
The heap being at 4GB is all the higher I would raise it, as you say GC
becomes costly with big heaps. The memory usage ramps up quite quickly
to well above the configured max heap within minutes. It comes up to
about 5.8GB of usage quickly as we manage many resources on many nodes.
We do not have many environments. We normally have a production branch
and only use a preprod branch to move changes up our environment with a
module that managed the puppet.conf on our nodes. We will keep looking,
but I don't see a smoking gun. Anyone else have any ideas? Puppet 4
was able to handle this load with only 4 JRuby workers and 4 Puppet
servers. Thanks for your help,
Mike
I suppose you have read this:
https://puppet.com/docs/puppetserver/6.0/tuning_guide.html
Read your description again. Think it is of value to look at the
stacktrace you get when the timeout occur. This to figure out what it is
that is timing out.
Best,
- henrik
On Monday, February 11, 2019 at 2:06:11 PM UTC-6, Henrik Lindberg wrote:
On 2019-02-11 14:42, Mike Sharpton wrote:
> Hey all,
>
> We have recently upgraded our environment from Puppetserver 4.2.2 to
> Puppetserver 6.0.2. We are running a mix of Puppet 4 and Puppet 6
> agents until we can get them all upgraded to 6. We have around 6000
> nodes, and we had 4 Puppetservers, but we added two more due to
capacity
> issues with Puppet 6. The load is MUCH higher with Puppet 6. To
the
> question, I am seeing longer and longer agent run times after
about two
> days of the services running. The only error in the logs that
seems to
> have any relation to this is this string.
>
> 2019-02-11T04:32:28.409-06:00 ERROR [qtp1148783071-4075] [p.r.core]
> Internal Server Error: java.io.IOException:
> java.util.concurrent.TimeoutException: Idle timeout expired:
30001/30000 ms
>
>
> After I restart the puppetserver service, this goes away for
about two
> days. I think Puppetserver is dying a slow death under this load
(load
> average of around 5-6). We are running Puppetserver on vm's that
are
> 10X8GB and using 6 Jruby workers per Puppetserver and a 4GB heap. I
> have not seen any OOM exceptions and the process never crashes. Has
> anyone else seen anything like this? I did some Googling and didn't
> find a ton of relevant stuff. Perhaps we need to upgrade to the
latest
> version to see if this helps? Even more capacity? Seems silly.
Thanks
> in advance!
>
There may be a slow memory leak that over time makes the server busy
win
non productive work (scanning for garbage on an ever increasing heap).
If you were to increase capacity you would risk only changing the 2
days
to a couple more, but not actually solving the issue.
Try to look at server memory usage over the two days.
Also, naturally, upgrade to latest and make sure modules are updated as
well.
Do you by any chance have many environments with different versions of
ruby code? The environment isolation "puppet generate types" may be of
help if that is the case as loaded ruby resource types become sticky in
memory.
- henrik
> Mike
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to puppet-users...@googlegroups.com <javascript:>
> <mailto:puppet-users+unsubscr...@googlegroups.com <javascript:>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com>
>
<https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/puppet-users/197c0ad5-83c0-4562-833b-82028f0e3e9c%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
<http://puppet-on-the-edge.blogspot.se/>
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/59932f88-0416-45d3-b938-0f3c417051fa%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-users/59932f88-0416-45d3-b938-0f3c417051fa%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/q3svuh%24pg9%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.
