What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-18 Thread jamesTheCruncher
I have a producer which sends persistent messages in /batches/ to a queue leveraging JMS transaction. I have tested and found that /Producer Flow Control/ is applied when using a batch size of 1. I could see my producer being throttled as per the memory limit I have configured for the queue. Here'

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-19 Thread jamesTheCruncher
Hi Tim, Thank you very much for the detailed response. Yes, I am aware that PFC will kick in eventually and do a collateral damage while using async sends without configuring /Producer Window Size/. However, I am not seeing that collateral damage when sending messages in batches with batch size >

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-19 Thread jamesTheCruncher
Hi Tim, The PFC documentation is very clear that PFC won't kick in for async sends unless /Producer Window Size/ is configured. / Producers that use Async Sends - generally speaking, producers of non-persistent messages - don't bother waiting for any acknowledgement from the broker; so, if a memo

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-19 Thread jamesTheCruncher
Could this be a bug? Or am I missing something? -- View this message in context: http://activemq.2283324.n4.nabble.com/What-is-the-correct-way-to-throttle-ActiveMQ-producers-who-send-persistent-messages-in-batches-to-a--tp4701204p4701233.html Sent from the ActiveMQ - User mailing list archive a

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-20 Thread jamesTheCruncher
Hi Tim, Thanks once again for taking the time to help me out. I really appreciate that. I'm very sorry for not mentioning the ActiveMQ version that I am using. I'm using ActiveMQ 5.10.0. Though my concern is about /batch send/ and PFC, I also tested async send to answer your questions. I sent aro

Re: AMQ pauses sending to consumers

2015-08-23 Thread jamesTheCruncher
Hi Richard, You told that a consumer consumes messages from a queue and then messages are published to secondary queues. Kindly note that PFC will be applied to this publisher. Are the messages published synchronously or asynchronously to the secondary queues? -- View this message in context:

Re: User Memory Issues

2015-08-24 Thread jamesTheCruncher
Hi Daniel, I guess you are getting the exception in your producer and you are sending messages asynchronously. Is this right? It would be helpful if you share your activemq.xml configuration here. -- View this message in context: http://activemq.2283324.n4.nabble.com/User-Memory-Issues-tp47011

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-24 Thread jamesTheCruncher
Hi Tim, I couldn't even send 1000 messages using jms.asyncSend=true (my payload being 2k) with the storeLimit of 2 mb. And there is no hint whatsoever about the per-destination memory limit of 1 mb I set. Here are the logs for your reference. And here's my activemq configuration (I'm sharing on

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-24 Thread jamesTheCruncher
I tested again with the storeUsage limit set to 32 mb (Please have a look at my previous reply to learn about the results I got with 2 mb as storeUsage). Not much difference. The producer is stopped on hitting the storeUsage limit. And here's my relevant activemq configuration: >From this may

Re: setProducerWindowSize inhibits propagation of ResAllocEx + producer stays forever blocked once it blocks

2015-08-25 Thread jamesTheCruncher
Can someone explain this please? This question was asked in 2011 and yet I see a similar behaviour with ActiveMQ 5.10.0. The producer never recovers after the ResourceAllocationException. There is very little or no documentation related to Producer Window Size. It would be really helpful to the com

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-25 Thread jamesTheCruncher
I just realized I was using ActiveMQ 5.8.0 for my tests. I really thought I was using v5.10.0. And little did I realize that the latest available version is v5.12.0. I'll rerun my tests with the latest ActiveMQ version and come back to you all. -- View this message in context: http://activemq.

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-26 Thread jamesTheCruncher
Sure I must have been living in a cave. Sorry for all the trouble Tim. As you mentioned, the producer throttling works out of the box while sending messages in batches. I believe even async sends should be throttled out of the box without bothering about Producer Window Size. This should work well

Re: setProducerWindowSize inhibits propagation of ResAllocEx + producer stays forever blocked once it blocks

2015-08-26 Thread jamesTheCruncher
This, I believe has been fixed in 5.9.0. I tested using the latest versions and found this to be working correctly. -- View this message in context: http://activemq.2283324.n4.nabble.com/setProducerWindowSize-inhibits-propagation-of-ResAllocEx-producer-stays-forever-blocked-once-it-blocs-tp3782

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-26 Thread jamesTheCruncher
Hi Tim, Yes it's kicking in for the per-destination limit. It will be great if you can edit the PFC wiki. But just one question. Is ProducerWindowSize still relevant? -- View this message in context: http://activemq.2283324.n4.nabble.com/What-is-the-correct-way-to-throttle-ActiveMQ-producers

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-08-31 Thread jamesTheCruncher
Hi Tim, Thanks for your continued interest in this subject. Yes, as per my testing, I didn't configure any ProducerWindowSize but found that the PFC was kicking in based on the per-destination limit I configured. I don't think it was the whole-connection PFC because I had set fairly larger limits

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-09-01 Thread jamesTheCruncher
Hi Tim, Thank you very much for the detailed explanation. I agree it is not as such bad to see "Producer stopped" message immediately after "Producer throttled". I merely registered the observations from my test which had no ProducerWindowSize configured. Now that my doubts are cleared, let me tr

Re: What is the correct way to throttle ActiveMQ producers who send persistent messages in batches to a queue?

2015-09-07 Thread jamesTheCruncher
Here is my finding. With a producer window size of 1 mb (same as my per-destination memory limit) and a payload of 2 kb both producer and consumer are eventually stopped after sending fewer than 2k messages while using a batch size > 1. When I used 100 as the batch size, I get the following excepti