Peter, You said no work is going to postgresql but how did you determine that? Are you able to see if PuppetDB (java) or postgres is consuming the CPU? How is disk activity?
Slow command processing is often correlated with long database queries -- it would be useful to turn on slow query logging for postgres (log_min_duration_statement) and watch for anything taking longer than a second or two. The guide here: https://docs.puppet.com/puppetdb/4.0/pdb_support_guide.html#puppetdb-4.0:-support-and-troubleshooting-guide has some additional information that is mostly relevant to your version in case it's useful. If you see some slow queries, report back and we can go from there. Some other questions: * Was the setup working fine up to when you posted this? How long has it been running? * Have you changed anything in your infrastructure that can be correlated to this? Sudden onset of this kind of thing is often due to introduction of e.g inlined file content if that helps * what does your PDB setup look like (how many machines, what kinds of machines, how much RAM, etc). Wyatt On Wed, Jun 28, 2017 at 12:54 PM, Mike Sharpton <sharpt...@gmail.com> wrote: > Hmm, I had thought these were one in the same. I don't have a 3 > environment to look at anymore. Good luck. > > On Wednesday, June 28, 2017 at 2:37:35 PM UTC-5, Peter Krawetzky wrote: >> >> I looked at both documents and the second one references the scheduler >> log files filling up. Mine are actually in the KahaDB directory. >> >> On Wednesday, June 28, 2017 at 12:25:57 PM UTC-4, Peter Krawetzky wrote: >>> >>> Last Sunday we hit a wall on our 3.0.2 puppetdb server. The cpu spiked >>> and the KahaDB logs started to grow eventually almost filling a >>> filesystem. I stopped the service, removed the mq directory per a >>> troubleshooting guide, and restarted. After several minutes the same >>> symptoms began again and I have not been able to come up with a puppetdb or >>> postgresql config to fix this. >>> >>> We tried turning off storeconfig in the puppet.conf file on our puppet >>> master servers but that doesn't appear to have resolved the problem. I >>> also can't find a good explanation as to what this parameter really does or >>> does not do even in the puppet server documentation. Anyone have a better >>> insight into this? >>> >>> Also is there a way to just turn off puppetdb? >>> >>> I've attached a file that is a snapshot of the puppetdb dashboard. >>> >>> Anyone experience anything like this? >>> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-users/06d2078d-120e-46b7-8f60-006582a39d47%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/06d2078d-120e-46b7-8f60-006582a39d47%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJDiH3F0U-uwdmkmNCA0VYUL%3Dc8ab0rm%2Bv_K20JMTUcoDUKzrA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
