I checked with IT guys and the disk I/O is not the issue, it can handle the data rate just fine.
I did not have time in finding a way to slow the producers. Building from 5.15.x/3 branch from source is not an option at this time. I also believe [AMQ-6815 KahaDB checkpoint needs to fail fast in the event of journal write failure] is not the culprit, althrough it might explain some of the other errors that I see in the activemq.log https://issues.apache.org/jira/browse/AMQ-6815 Does ActiveMQ baseline have any load test that maybe we can simulate this scenario: e.g. send decent amount of messages to a non-persistent queue (with a relatively small JVM max heap) to test this under ActiveMQ 5.14.x vs 5.15.x? That would be better to have some type of regression test for this scenario inside ActiveMQ source code. I have more info from another experiment I did today. I changed systemUsage.memoryUsage to 1%. I see an INFO Usage Manager Memory Limit .. reached on queue://... Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it... a couple times. Then ~200ms afterward, the tmp_storage initialized INFO log. On the web console, I see Memory percent used to be 99% and 100% and Temp percent used 3%. Eventually, the broker starting having errors in the Async Writer Thread Shutdown. I did a similar test under ActiveMQ 5.14.5 broker (memoryUsage is at default 70%), and here the Memory used says under 70%, of particular note is that the 5.14.5 broker Memory percent used drops alot, from 70% to 25% when it flushes non-persistent message to the Temp storage (I can see Temp used increase). I recall a similar scenario running under ActiveMQ 5.12.2, where the Memory used decreases dramatically when it flushes non-persistent message to the Temp storage on disk. Regards, Neon -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
