Hello,
We are facing a really big problem in production. From time to time, the
broker stops delivering the messages to some consumers which are located on
the same machine as the broker. After a restart, the broker delivers some
more messages but new messages can accumulate. The only way to fix the
problem for another 2 - 3 weeks is to clear the kahaDB, but this shouldn't
be happening in production.
Finally we managed to get a copy of the kahaDB before the support team
restarted the broker, and when I deployed it into my broker, I received the
following error at broker initialization:
2014/07/10 15:57:15,033 | INFO | WrapperSimpleAppMain | MessageDatabase
| ahadb.MessageDatabase$Metadata 155 | KahaDB is version 4
2014/07/10 15:57:16,502 | ERROR | WrapperSimpleAppMain | BrokerService
| .activemq.broker.BrokerService 555 | Failed to start ActiveMQ JMS Message
Broker (fpskylla_activemq, null). Reason: java.io.IOException: Invalid
location: 1193:28702079, : java.lang.NegativeArraySizeException
java.io.IOException: Invalid location: 1193:28702079, :
java.lang.NegativeArraySizeException
at
org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
at org.apache.kahadb.journal.Journal.read(Journal.java:601)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:898)
at
org.apache.activemq.store.kahadb.MessageDatabase.recoverProducerAudit(MessageDatabase.java:577)
at
org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:504)
at
org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:360)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:408)
at
org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:237)
at
org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:176)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
at
org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:188)
at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:512)
at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1612)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1553)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1483)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:285)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:285)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:608)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at org.apache.servicemix.Main.main(Main.java:69)
------------------------------------------------------------
The broker initialization fails and everything stops. If I restart the
broker, everything goes ok and I see the following logging:
2014/07/10 16:06:53,330 | INFO | WrapperSimpleAppMain | BrokerService
| .activemq.broker.BrokerService 508 | Using Persistence Adapter:
KahaDBPersistenceAdapter[d:\activemq\apache-servicemix-3.6.0-fuse-00-89\activemq-data]
2014/07/10 16:06:53,963 | INFO | WrapperSimpleAppMain | MessageDatabase
| ahadb.MessageDatabase$Metadata 155 | KahaDB is version 4
2014/07/10 16:06:55,387 | INFO | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 511 | Recovering from the journal ...
2014/07/10 16:06:55,393 | DEBUG | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 1212 | message not found in sequence id
index: ID:FR0502-1079-1398479702307-13:1:1:1:832
2014/07/10 16:06:56,902 | DEBUG | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 1212 | message not found in sequence id
index: ID:FR0502-1143-1403750130523-13:1:1:1:696
2014/07/10 16:06:57,552 | DEBUG | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 1230 | no message sequence exists for id:
ID:FR050204-1068-1404358678116-2:1:53:1:9 and sub:
PosTransaction_necExportService:PosTransaction_necExportService
2014/07/10 16:06:57,552 | DEBUG | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 1230 | no message sequence exists for id:
ID:FR050204-1068-1404358678116-2:1:53:1:9 and sub:
eReSdcBS-receiver_FR0502:eReSdcBS-receiver_FR0502_eres-postransaction-01-0502-@
2014/07/10 16:06:58,088 | DEBUG | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 1212 | message not found in sequence id
index: ID:FR0502-1280-1404441548401-13:1:1:1:896
2014/07/10 16:06:58,527 | INFO | WrapperSimpleAppMain | MessageDatabase
| q.store.kahadb.MessageDatabase 524 | Recovery replayed 33353 operations
from the journal in 3.152 seconds.
2014/07/10 16:06:58,542 | INFO | WrapperSimpleAppMain | BrokerService
| .activemq.broker.BrokerService 785 | ActiveMQ 5.5.1.fuse-70-097 JMS
Message Broker (fpskylla_activemq) is starting
I spent a lot of time investigating this problem and I came across some
known bugs (AMQ-3104, AMQ-3422, AMQ-3775, AMQ-3806, MB-1046), but I have the
fixes in the ActiveMQ version that I am using. I am very confused by the
fact that my ActiveMQ version, 5.5.1.fuse-70-097, contains fixes which
should be found in the ActiveMQ 5.6.0. Also, the prefetchPolicy has been set
to 1 and the auto acknowledge mode is on.
Do you think that a thread dump would help me? Any idea is really
appreciated because we are stuck.
If somebody can analyze the kahaDB, I would be happy to zip it and send it.
Many thanks,
Cristian.
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Another-NegativeArraySizeException-problem-ActiveMQ-5-5-1-fuse-70-097-tp4683082.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
