First, the categorical statement is that when you hit an Out of memory error, you have no guarantee that your application is still in a valid state, so you should terminate the JVM and restart. So my first question is, does this happen when you haven't run the JVM out of heap space? If not, the behavior you're seeing isn't a bug. And if this occurred in an operational broker and you haven't restarted it yet, that should be your first order of business.
With that said, I'll try to address your other questions, in case the behavior you're seeing turns out to not be related to the OOM you saw. I agree with your statement that in your case, messages were not discarded because they expired, since a 0 value for the expiration means that feature's not in use for this mesages. One way for non-expired messages to be discarded is to use the strategies listed on http://activemq.apache.org/slow-consumer-handling.html. Are you applying any of those configurations? (Since neither you nor Mike posted your full config files, it's hard to know what you're using and what you're not.) Those strategies don't result in the creation an actual queue data structure that run out of space, but they do allow the application of limits on the number of messages that can be pending for a given consumer. So the specific situation you suggested isn't possible, but there are code paths that will discard messages when there are too many. However, those are only enabled if you explicitly configure them, hence the question about how you've configured the broker. On a different note, from what you said, you're statically including a virtual topic, which will cause its messages to be sent to the remote broker irrespective of whether there are any consumers there. So, for the topic in question, were there any consumers for that topic on the remote broker? Could this have been as simple as a message being discarded because there were no consumers once the message arrived on the remote broker? For normal topics, messages discarded for that reason wouldn't go to the DLQ, but I've not used virtual topics so I don't know whether that's a possibility with them, so this question might be irrelevant. Tim On Nov 17, 2017 4:11 AM, "augustl" <aug...@augustl.com> wrote: > I also got these messages when I had a network connector setup. The setup > seems similar to OP, where we had a message configured to be sent via a > staticallyIncludedDestinations block, and it ended up in the DLQ with > "TopicSubDiscard". Here's a full dump of one of the messages that ended up > in the DLQ: > > <http://activemq.2283324.n4.nabble.com/file/t378787/ > Screenshot_from_2017-11-17_12-05-12.png> > > As you can see, message expiration was set to 0. So it seems like there was > something else that casued it to expire. > > I noticed in the source code that there's another way for this error to > occur: > > https://github.com/apache/activemq/blob/08aa5118ffcd8dfc4ec813d799a61e > f1ee4421d1/activemq-broker/src/main/java/org/apache/ > activemq/broker/region/TopicSubscription.java#L189 > > Seems like some sort of discarding of old messages when consumption is > slow? > > Why does this happen to messages sent to a topic? Is there some sort of > internal queue that's created for delivering messages over a network > connector, and maybe this queue filled up somehow? > > By the way, we also got a lot of these in the logs of the activemq with the > network connector where the message originated from and where it ended up > in > the DLQ: > > 2017-11-17 07:04:21,750 | ERROR | Checkpoint failed | > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal > Checkpoint Worker > java.lang.OutOfMemoryError: Java heap space > 2017-11-17 07:04:21,750 | INFO | Ignoring no space left exception, > java.io.IOException: Java heap space | > org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal > Checkpoint Worker > java.io.IOException: Java heap space > at > org.apache.activemq.util.IOExceptionSupport.create( > IOExceptionSupport.java:40)[activemq-client-5.14.4.jar:5.14.4] > at > org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run( > MessageDatabase.java:433)[activemq-kahadb-store-5.14.4.jar:5.14.4] > at > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511)[:1.8.0_144] > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_ > 144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor. > java:180)[:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run( > ScheduledThreadPoolExecutor.java:294)[:1.8.0_144] > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149)[:1.8.0_144] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624)[:1.8.0_144] > at java.lang.Thread.run(Thread.java:748)[:1.8.0_144] > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
