In case it helps anyone else, there is likely no need to fiddle with the Heap etc. The root cause in my case was somehow during an upgrade the master was unpinned from the "Pe Certificate Authority" nodegroup. I've seen it on a few boxes that were upgraded using nc_migrate tool FWIW.
On Friday, July 28, 2017 at 3:45:44 PM UTC-7, Mike Jeski wrote: > > cheers Ryan, I have read the guide and made changes, we'll see how it goes. > > On Friday, July 28, 2017 at 11:27:29 AM UTC-7, Ryan Whitehurst wrote: >> >> On Fri, Jul 28, 2017 at 9:01 AM, Mike Jeski <jesk...@gmail.com> wrote: >> >>> Cheers mate. This version of PE is actually running java 7. >>> >> >> Ah, well in that case bumping up the MaxPermSize will have an effect, but >> the problem is still almost certainly caused by too low of a heap size, so >> you should still read the information here about tuning the JVM heap size >> and set the -Xmx value higher: >> >> https://docs.puppet.com/puppetserver/1.2/tuning_guide.html >> >> >>> >>> On Thursday, July 27, 2017 at 1:19:59 PM UTC-7, Mike Jeski wrote: >>>> >>>> Hi group, >>>> >>>> running PE 3.8.7 (while cleaning up code for upgrade to puppet 4) >>>> seeing HttpClientException constantly in the puppetserver.log. >>>> >>>> WARN [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true >>>> com.puppetlabs.http.client.HttpClientException: Request cancelled >>>> >>>> Happens every few minutes, reports continue to process and then >>>> eventually I see this: >>>> >>>> WARN [o.e.j.s.HttpChannel] /production/certificate/ca?fail_on_404=true >>>> java.lang.OutOfMemoryError: unable to create new native thread >>>> >>>> after which (most of the time) agent runs & reports are processed, >>>> however not always. Sometimes reports stop processing altogether and the >>>> console shows all 120 nodes as unresponsive. >>>> >>>> Where to start debugging this? >>>> >>>> What I've done so far is in /etc/sysconfig/pe-puppetserver I've >>>> modified the JAVA ARGS >>>> MaxPermSize is now 512m up from 256m. And I've added -XX:+UseG1GC >>>> >>>> The Machine is RHEL6 (VMWARE) 16GB RAM & 8Cores. r >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-dev+...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-dev/81eea74d-a217-4cf7-ac97-f0cab1061ddb%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/puppet-dev/81eea74d-a217-4cf7-ac97-f0cab1061ddb%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/a5eb56cf-875a-410c-b00a-424a84371fd4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
