Hey all, Update to this thread in case others have a similar issue. It IS the setting below it appears. This is listed as a "Potential" tuning setting. It is basically required for any real workload. I saw this after pulling out some of last hairs I have left on my head (in messages, which I was not even focused on, Doh!).
Feb 15 08:38:33 puppetserver: OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Feb 15 08:38:33 puppetserver: OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= Feb 15 08:38:33 puppetserver: CodeCache: size=245760Kb used=207748Kb max_used=242118Kb free=38011Kb Feb 15 08:38:33 puppetserver: bounds [0x00007f4394be4000, 0x00007f43a3be4000, 0x00007f43a3be4000] Feb 15 08:38:33 puppetserver: total_blobs=29412 nmethods=28684 adapters=635 Feb 15 08:38:33 puppetserver: compilation: disabled (not enough contiguous free space left) Boom goes the dynamite. This explains a great many things. I have just adjusted the settings and now need to wait and see if it is better. I will drop a final post to finish this thread if it works out. Mike On Tuesday, February 12, 2019 at 7:24:21 AM UTC-6, Mike Sharpton wrote: > > I just found this as well. > > If you're working outside of lab environment, increase ReservedCodeCache > to 512m under normal load. If you're working with 6-12 JRuby instances > (or a max-requests-per-instance value significantly less than 100k), run > with a ReservedCodeCache of 1G. Twelve or more JRuby instances in a > single server might require 2G or more. > > We need to look into this setting. > > On Monday, February 11, 2019 at 7:42:01 AM UTC-6, Mike Sharpton wrote: >> >> Hey all, >> >> We have recently upgraded our environment from Puppetserver 4.2.2 to >> Puppetserver 6.0.2. We are running a mix of Puppet 4 and Puppet 6 agents >> until we can get them all upgraded to 6. We have around 6000 nodes, and we >> had 4 Puppetservers, but we added two more due to capacity issues with >> Puppet 6. The load is MUCH higher with Puppet 6. To the question, I am >> seeing longer and longer agent run times after about two days of the >> services running. The only error in the logs that seems to have any >> relation to this is this string. >> >> 2019-02-11T04:32:28.409-06:00 ERROR [qtp1148783071-4075] [p.r.core] >> Internal Server Error: java.io.IOException: >> java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms >> >> >> After I restart the puppetserver service, this goes away for about two >> days. I think Puppetserver is dying a slow death under this load (load >> average of around 5-6). We are running Puppetserver on vm's that are >> 10X8GB and using 6 Jruby workers per Puppetserver and a 4GB heap. I have >> not seen any OOM exceptions and the process never crashes. Has anyone else >> seen anything like this? I did some Googling and didn't find a ton of >> relevant stuff. Perhaps we need to upgrade to the latest version to see if >> this helps? Even more capacity? Seems silly. Thanks in advance! >> >> Mike >> > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2aa8d37-609f-42ab-b04d-c497abfc699a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
