Hi Guillaume, Can you post the output of
rpm -qa | grep puppet ? It looks as if your puppetdb-terminus version could be out of sync with that of puppetdb. Wyatt On Wed, Sep 3, 2014 at 7:17 AM, Guillaume Blairon <g...@yom.be> wrote: > Hi, > > We're using puppetdb 2.1.0 and puppet 3.6.2 on CentOS. PuppetDB has been > working fine for a few weeks. > > Yesterday we added a new puppet master to handle our ~400 servers. When we > added the new master in the SRV records, PuppetDB began to show an erratic > behaviour : > - the number of iops on /var has raised from 50/sec to 500+/sec > - the /var/lib/puppetdb/mq/localhost/scheduler is being filled with > db-*.log files until filesystem gets full > - the logs are filled with this type of messages (~1 per second) : > --- > 2014-09-03 15:48:51,006 ERROR [c.p.p.command] > [24506b0f-6fcd-4009-9761-bfe3652ee1bd] [replace facts] Retrying after > attempt 5, due to: java.lang.IllegalArgumentException: No method in > multimethod 'process-command!' for dispatch value: ["replace facts" 3] > java.lang.IllegalArgumentException: No method in multimethod > 'process-command!' for dispatch value: ["replace facts" 3] > at clojure.lang.MultiFn.getFn(MultiFn.java:160) ~[puppetdb.jar:na] > at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$produce_message_handler$fn__10782.invoke(command.clj:630) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731$fn__10735.invoke(command.clj:537) > ~[na:na] > at > com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown > Source) ~[na:na] > at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731.invoke(command.clj:536) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716$fn__10717.invoke(command.clj:490) > ~[na:na] > at > com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown > Source) ~[na:na] > at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716.invoke(command.clj:489) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__10726.invoke(command.clj:512) > [na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_meter$fn__10706.invoke(command.clj:450) > [na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__10740.invoke(command.clj:552) > [na:na] > at com.puppetlabs.mq$create_message_listener$reify__9909.onMessage(mq.clj:270) > [na:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) > [puppetdb.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > --- > > Until now, the only solution we have found is to stop puppetdb, > remove /var/lib/puppetdb/mq/localhost and restart puppetdb. But it's only > buying us a few hours before the filesystem gets full again. > > I'm not actually sure that the problem has a relation with the new master, > but that's more or less when the problem started. Do you have an idea of > what's happening here ? Is there something else I could check ? > > Cheers, > -- > Guillaume > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJDiH3EOVaQEYNOKkMz3QDCU%3Dhz2%2BrCzPn30kGQ%2BKoTWOwAOLw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
