On Tuesday, January 22, 2013 9:19:02 AM UTC-6, Daniel wrote: > > On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber > <k...@puppetlabs.com<javascript:>> > wrote: > >> This sounds like a sensible workaround, I will definitely have a look. > I > >> haven't yet had enough time to look at the issue properly, but it seems > that > >> this very long time is indeed consumed by catalog construction. > Puppetdb > >> fails after this is finished, so it seems that it dies when nagios host > >> tries to report its catalog back. > > > > Do you mean it dies from an OOM when it tries to report the catalogue > back? > > Yes, that's what it looks like. Of course I can prevent it by giving > it more memory (which I did), but I already have postgres backed > puppetdb and had to give puppetdb 3GB, or a puppet agent run on a > single host (OK, with thousands of exported resources to collect and > process) that takes about 70 minutes can still kill it. This waiting > 70 minutes for it to die is an insult to injury... Overall not great. > I'm happy to redo this setup if I'm doing something wrong, but it just > seems like this is exponential (30-odd nodes - 2 minutes, 100-odd > nodes, 70 minutes). > > This sounds like a memory-management problem. If you're running up against (or close to) a memory limit, then the Java-based puppetdb will start doing a lot of garbage collection as available heap memory becomes scarce. That will greatly slow down the whole Java VM, especially so if the GC runs don't free much memory. Also, if you give the VM more memory than available RAM then you will have slowdowns related to paging parts of the heap between physical and virtual memory; depending on memory access patterns, that could happen frequently and be very costly.
On the other hand, some versions of the Java VM have problems with large amounts of heap memory. One would hope that such problems have been resolved by now, but you could anyway try reducing the heap memory to about 1.5G. That's close to the maximum that you can safely use with some VM implementations. I certainly agree that puppetdb has a serious problem if it requires such large amounts of memory to handle resource collections with a few thousand elements. If roughly 3000 (at ~30 resources * ~100 nodes) exported resources require in excess of 3GB then that's in the vicinity of 1MB per resource -- absurd! On the other hand, that's *so* absurd that I suspect something else is going on here. It smacks of a combinatoric problem, either in your manifests or in the puppetdb implementation. There aren't enough data points to be sure, but your run times could be scaling with the square of the number of nodes. Is there any chance that the number of exported resources is scaling the same way? That would explain both time and memory consumption. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/25e0ER5_C4gJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
