> I'm @ken_barber on irc btw if that is easier. > Can't use IRC here, sorry. :(
> Okay. Did you clear the ActiveMQ queues after doing this? I usually > just move the old KahaDB directory out of the way when I do this. > I haven't though about myself, but it makes sense, so I just flushed the queue again while puppetdb service was stopped. Since this last restart it seems smooth, but it's only up for a few minutes. > Okay, so were those old queries that are taking 90 minutes, what are they? > > SELECT COUNT(*) AS c FROM certname_catalogs cc, catalog_resources cr, certnames c WHERE cc.catalog=cr.catalog AND c.name=cc.certname AND c.deactivated IS NULL 18 instances were running, oldest was ~90 mins at the time SELECT COUNT(*) AS c FROM (SELECT DISTINCT resource FROM catalog_resources cr, certname_catalogs cc, certnames c WHERE cr.catalog=cc.catalog AND cc.certname=c.name AND c.deactivated IS NULL) 12 instances were running, oldest was ~90 mins at the time > Have you tried shutting down the PuppetDB, clearing the queue - > running full vacuum on the database then starting it all up again? > Not yet, according to the Postgresql documentation full vacuum is not really recommended. Also it might take a while, and PuppetDB would be unavailable for the duration. > What version of PostgreSQL are we talking here? What distro and > version specifically? How big is the database itself? > 8.4.13 on a RHEL6 based distribution. The database size might be an indicator that something is not ok. It currently takes up 512Gbs, and it grew 100Gbs over a week. We made a database dump before the PuppetDB upgrade and it was ~1Gb uncompressed. The full vacuum might solve this, but my guess that it would run quite long. > How many nodes and how frequently are they checking in? A screenshot > of your puppetdb dashboard might give us a good snapshot of the size > of your environment as well. > We have ~2500 nodes, run interval for the two major environments: 60mins for 1593 machines and 30 mins for 496. Splay limit is 900sec for both. Two screenshots from today: http://imgur.com/kG5Sth5,bLFqGAX#0 and http://imgur.com/kG5Sth5,bLFqGAX#1 > Has any tuning been performed on your postgresql server? A copy of > your postgresql.conf would be interesting. > No tuning that I'm aware of. The same default postgresql.conf as deployed by the puppetlabs-postgres module. It's here: http://pastebin.com/njvGm4eu > > We can do this all privately if you are concerned about sharing > information publicly as well - up to you. Like I said, I'm also on > Freenode IRC as @ken_barber if this is better for you. > > ken. > So far I don't feel like I shared anything sensitive, so this might be useful for others as well. Thanks for the help! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
