Slow throughput after several hundred messages

Hi, I am using ActiveMQ 5.4.1. Our application runs on GlassFish application server 2.1.1, however I have reproduced the slow behaviour described below using a J2SE standalone client as well. Our application uses multiple stateless session beans to concurrently produce messages which are stored i

Re: Slow throughput after several hundred messages

I have just downloaded version 5.4.2 and run the same standalone j2se test case I developed to highlight this issue against this version, but I am still experiencing the same behaviour as with 5.4.1. -- View this message in context: http://activemq.2283324.n4.nabble.com/Slow-throughput-after-se

Re: Slow throughput after several hundred messages

I have attached a compressed rar file containing the broker debug trace: http://activemq.2283324.n4.nabble.com/file/n3085354/logs.rar logs.rar Due to attachment file size limits, I will attempt in separate posts to add the broker configuration and crude netbeans project I used to show this beh

Re: Slow throughput after several hundred messages

Here is the sample netbeans project that I have used to highlight the problem: http://activemq.2283324.n4.nabble.com/file/n3085358/StandaloneJMSWriteConsume.rar StandaloneJMSWriteConsume.rar It is very crude, simply run the ConcurrentJMSWriteConsumeJ2SETest class and it will continue to write a

Re: Slow throughput after several hundred messages

The activemq heap dump during execution of the sample test case: http://activemq.2283324.n4.nabble.com/file/n3085361/activemq-heapdump-during-test-after-5-mins.rar activemq-heapdump-during-test-after-5-mins.rar -- View this message in context: http://activemq.2283324.n4.nabble.com/Slow-throug

Re: Slow throughput after several hundred messages

Heapdump following killing the test application and performing GC: http://activemq.2283324.n4.nabble.com/file/n3085363/activemq-heapdump-after-test.rar activemq-heapdump-after-test.rar -- View this message in context: http://activemq.2283324.n4.nabble.com/Slow-throughput-after-several-hundred-

Re: Slow throughput after several hundred messages

Also, here is the broker configuration I used: http://activemq.2283324.n4.nabble.com/file/n3085451/activemq.xml activemq.xml -- View this message in context: http://activemq.2283324.n4.nabble.com/Slow-throughput-after-several-hundred-messages-tp3082431p3085451.html Sent from the ActiveMQ - Us

Re: Slow throughput after several hundred messages

I have tried each of the different cursor types with 5.4.1, but since downloading the 5.4.2 version, I have not changed the cursor used, so this is the default configuration, where no cursor is defined for queues - this should result in the default store cursor being used. I have just tried with

Re: Slow throughput after several hundred messages

I have left the test application running over night, it is still consuming and producing messages but is blocking very often now (every 15-20 seconds for around 10 seconds each time). I used visualVM to analyse the garbage collection, which seems to be running around once every two minutes or so

Re: Slow throughput after several hundred messages

Here is the activemq logs from a test run where I allowed it to begin processing, block once and waited for it to resume processing, where I have turned on verbose garbage collection output: http://activemq.2283324.n4.nabble.com/file/n3086805/logs_with_verbose_gc.rar logs_with_verbose_gc.rar --

Re: Slow throughput after several hundred messages

In order to try to work out just what is happening during these blocking periods, I have used visual vm to perform cpu profiling, take a heapdump and a thread dump when the broker becomes unresponsive. Here are the results: http://activemq.2283324.n4.nabble.com/file/n3086836/visual_vm_analysis_du

Re: Slow throughput after several hundred messages

Here is the gc.log and activemq logs, once again taken from a run where I allowed it to process, block and then continue processing: http://activemq.2283324.n4.nabble.com/file/n3087092/logs_verbose_gc.rar logs_verbose_gc.rar -- View this message in context: http://activemq.2283324.n4.nabble.c

Re: Slow throughput after several hundred messages

I believe the cause of the EOFException and the blocking behaviour and slowdown is caused by creating and closing connections for each read/write. In the standalone j2se application I attached in an earlier post, each thread creates a new connection for each request before closing down the resourc

Re: Slow throughput after several hundred messages

Thanks Reynald, that utility could be useful. I think I have already pinned down this problem to be opening and closing of connections to the broker. If I share a single or a couple of connections for all processing, I do not get the broker slowdown issue. Unfortunately, our application runs in Gl

Re: Slow throughput after several hundred messages

I am using a prefetch of 0. This has been configured in the ra.xml in the activemq resource adapter we use to connect up to glassfish. I configured this by adding "&jms.prefetchPolicy.all=0" to the connection url. Just for good measure I also added the following into the ra.xml as well:

Re: Slow throughput after several hundred messages

here is the ra.xml I use: http://activemq.2283324.n4.nabble.com/file/n3090748/ra.xml ra.xml -- View this message in context: http://activemq.2283324.n4.nabble.com/Slow-throughput-after-several-hundred-messages-tp3082431p3090748.html Sent from the ActiveMQ - User mailing list archive at Nabble

Re: Slow throughput after several hundred messages

I took a look at the XAPooledConnectionFactory, but I am struggling to understand how I could use this when connecting to the broker from a glassfish container via the activemq jca resource adapter. Is this configurable within the resource adapter or jca connection pool somehow? -- View this mess

Re: Slow throughput after several hundred messages

Just to note that I was able to resolve this issue by changing our code to maintain a single connection for the multiple concurrent processing threads. The slow down seems to be caused by continuous opening and closing of queue connections to ActiveMQ. -- View this message in context: http://acti