If you clear the queue and rollback to the original version does the problem disappear? If you're having processing problems at the latest version thats what I would do, as I presume we're talking production here right?
> Can this be somehow related to the the KahaDB leak thread? No - it doesn't sound like it. > ERROR: insert or update on table "certname_catalogs" violates foreign key > constraint "certname_catalogs_catalog_fkey" > DETAIL: Key (catalog)=(d1c89bbef78a55edcf560c432d965cfe1263059c) is not > present in table "catalogs". > STATEMENT: INSERT INTO certname_catalogs (certname,catalog,timestamp) VALUES > ($1,$2,$3) RETURNING * That is a bit of a concern, are you receiving a lot of these? Is this constant? > Use of unversioned APIs is deprecated; please use /v1/resources These are appearing in your slow query log - are you sure you're running puppetdb-terminus 1.1.1 as well? Did you restart your master after upgrading the terminus? ken. On Thu, Feb 28, 2013 at 12:54 PM, ak0ska <akos.he...@gmail.com> wrote: > The queue jumped back to ~1 600, and according to the puppetdb logs, it > hasn't processed a single entry for like 20 minutes, but the log shows a lot > of slow queries (see example below). Postgres logs show no error. > > And now it started processing the queue again. > > > Slow queries: http://pastebin.com/2B4azRib > > > > > On Thursday, February 28, 2013 1:09:15 PM UTC+1, ak0ska wrote: >> >> Or maybe it did. The queue size went up to like 900 and after a while it >> normalized and started slowly decreasing. Right now it's empty. >> There were 2 insert errors in postgres log since its restart. >> >> It was perhaps on our end, but it would be good to hear some theories, or >> suggestions where else to look for a problem in a similar case >> >> On Thursday, February 28, 2013 10:38:42 AM UTC+1, ak0ska wrote: >>> >>> We flushed the queue and also restarted Postgres. Didn't help. >>> >>> On Thursday, February 28, 2013 10:14:16 AM UTC+1, ak0ska wrote: >>>> >>>> Hello, >>>> >>>> We have upgraded Puppetdb from 1.0.2 to 1.1.1 last week, and recently we >>>> noticed that queue is not processed. We don't know if this problem is >>>> directly linked to the upgrade. There are now 25 000 messages waiting in >>>> the >>>> queue. We also see errors in the Postgres log: >>>> >>>>> ERROR: insert or update on table "certname_catalogs" violates foreign >>>>> key constraint "certname_catalogs_catalog_fkey" >>>>> DETAIL: Key (catalog)=(d1c89bbef78a55edcf560c432d965cfe1263059c) is not >>>>> present in table "catalogs". >>>>> STATEMENT: INSERT INTO certname_catalogs (certname,catalog,timestamp) >>>>> VALUES ($1,$2,$3) RETURNING * >>>> >>>> >>>> Can this be somehow related to the the KahaDB leak thread? >>>> (https://groups.google.com/d/topic/puppet-users/gWACi4a0BZs/discussion ) >>>> >>>> Any thoughts? Would flushing the queue solve the problem? >>>> >>>> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
