We have a very strange problem with our ActiveMQ broker that happens when we restart it after running it for several months. We have not been able to reproduce it in-house. It only happens in production at our customer's site. At this time we have seen this issue at least three times.
What happens is that some very old (~1 month) messages come back from the dead when we restart the broker. The last time this happened we saw ~1600 resurrected messages. We can see in our logs that these messages have already been processed. There have been more messages written to the same queue, but these did not come back to life. At the time of broker shutdown the problematic queue was empty with no consumers or producers connected (they were stopped before the broker was stopped). We tried deleting the kahadb directory when this happened the last time, but that didn't help. The log does not contain any errors during startup. We get a couple of rows similar to "Recovery replayed 170629 operations" in the log, but this is normal as far as I know. If I am wrong about this being normal, please let me know. --- We have a mix of applications that communicate using an ActiveMQ broker. The Java applications, like Wildfly, use the OpenWire protocol. The C++ applications use the AMQP protocol, with "transport.transformer=jms" setting in "activemq.xml". The C++ applications use transactional sessions to consume messages from the broker. The problematic queues are consumed by one of these C++ applications. The messages are produced by one of the Java/Wildfly applications. Most, but not all, messages that are sent through the broker are non-persistent. The messages on the problem queues are persistent. We use queues for everything. Topics are not used. There is only a single ActiveMQ broker instance. ActiveMQ version: 5.14.5. --- We have tried to trigger this error in-house, but without any success. We have only seen it in our customer's production environment. Our own hypothesis is that the kahadb data file that contains the acks is deleted, but not the file containing the actual messages. After restart it then looks like the messages were never sent. Even if this is true we have no idea how it can happen. All our manual tests show that ActiveMQ does the right thing. There must be some special case that we cannot find. Does anyone have an idea what could be the cause of this issue? Has anyone seen anything like this before? Any ideas on how get a reproducible case? -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
