Hi, A question about Puppetserver performance.
For quite a while now, our primary Puppet server is suffering from severe slowness and high CPU usage. We have tried to tweak its settings, giving it more memory (Xmx = 6 GB at the moment) and toying with the 'max-active-instances' setting to no avail. The server has 8 virtual cores and 12 GB memory in total, to run Pupperserver, PuppetDB and PostgreSQL. Notably, after a restart, the performance is acceptable for a while (several hours, up to a almost day), but then it plummets again. We figured that the server was just unable to cope with the load (we had over 270 nodes talking to it in 30 min intervals), so we added a second master that now takes more than half of that load (150 nodes). That did not make any difference at all for the primary server. The secondary server however, has no trouble at all dealing with the load we gave it. In the graph below, that displays catalog compilation times for both servers, you can see the new master in green. It has very constant high performance. The old master is in yellow. After a restart, the compile times are good (not great) for a while.The first dip represents ca. 4 hours, the second dip was 18 hours. At some point, the catalog compilation times sky-rocket, as does the server load. 10 seconds in the graph below corresponds to a server load of around 2, while 40 seconds corresponds to a server load of around 5. It's the Puppetserver process using the CPU. The second server, the green line, has a consistent server load of around 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2 cores (it's an EC2 t3.medium). If I have 110 nodes, doing two runs per hour, that each take 30 seconds to run, I would still have a concurrency of less than 2, so Puppet causing a consistent load of 5 seems strange. My first thought would be that it's garbage collection or something like that, but the server plenty of memory (OS cache has 2GB). Any ideas on what makes the Puppetserver starting using so much CPU? What can we try to keep it down? Thanks, Martijn Grendelman -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0f6e7373-404a-45fd-8bc7-5daed3fa67f3%40googlegroups.com.
