ActiveMQ - TempTopic/TempQueue makes MQ frozen during high traffic flow
Hello, Here is ActiveMQ 5.3.0 with clients connected using openwire and CMS activemqcp-3.0.1. Server is Intel Platform with 8GB RAM : 1) policyEntry for topic=">" and queue=">" set up to memoryLimit=20mb and producerFlowControl="false" 2) memoryUsage limit="256 mb", storageUsage = 10gb, tempUsage = 10gb 3) transport openwire with ?wireFormat.maxInactivityDuration=0 and we use quite a lot TempTopic's and TempQueues ... what I found after really heavy traffic that : 1) Heap memory usage in JMX console went above 256MB and ActiveMQ was frozen -> 2) in the console such messages appeared: INFO | Started selectchannelconnec...@0.0.0.0:8161 INFO | Slow KahaDB access: Journal append took: 511 ms, Index update took 0 ms INFO | Slow KahaDB access: Journal append took: 542 ms, Index update took 0 ms INFO | Usage Manager memory limit reached on temp-queue://42455840-0bb5-0101-49b1-304ecf24dbbd:2. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info INFO | Usage Manager memory limit reached on temp-queue://924b8029-97ad-f8e2-32ff-52384d24558e:0. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info I also confirmed in JMX console that for Topic and Queue producerFlowControl is disabled but for TempTopic and TempQueue is enabled. My question is: 1) is it possible to disable producerFlowControl for TempTopics and TempQueues? 2) maybe clients apps using CMS (cpp-3.0.1) do not do kind of "confirmations" to Broker? 3) what can be done to improve this set up: of course memoryUsage can be changed I mean set bigger but I'm really worried about this memory being consumed and not released. Any comments on that? ;) Many thanks, Marcin -- View this message in context: http://old.nabble.com/ActiveMQ---TempTopic-TempQueue-makes-MQ-frozen-during-high-traffic-flow-tp27534454p27534454.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
AMQ (5.3.0) with CMS-3.01 -> consumes all memory and stops clients and producers
Hello here is what we have:
1) AMQ -5 3.0 with the following configuration:
http://activemq.apache.org/schema/core";
brokerName="localhost" useJmx="true" dataDirectory="${activemq.base}/data">
System is being run with Xm=2048M
2) Producers and clients use CMS - 3.0.1 - we don't have any features added
to CreateFactory - all defaults are used.
Results:
1) during the day watch MQ via JMX I have: totalEnqueueCount=1193470,
totaldequeuecount=628130,totalmessagecount=1052834 - this I think is wrong
... shouldn't be like that: totalmessage+totaldequeue = totalenqueue?
2) memorypercentusage=73 and it's counting ... once it gets to 100% all is
frozen and in log file I get messages like this: INFO | Usage Manager memory
limit reached on temp-queue://42455840-0bb5-0101-49b1-304ecf24dbbd:2.
Producers will be throttled to the rate at which messages are removed from
this destination to prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info
quite strange as flow-control is disabled.
3) watching Heap memory Usage I can see "saw chart" but it's going up and up
to 1GB and then is saturated (AMQ is frozen at that time).
4) the only thing I can do is to stop AMQ and restart ... however on
production is not a solution.
Maybe I have something wrong with configuration file? I'm also a little bit
confused about memoryLimit in topic/queue policy and memoryUsage. What
exactly these options do? I thought that if memoryUsage is 1GB and
percentUsage is 70% all messages are dropped into "file" and memory is
clean. Maybe I mixed this all up and it's not as I think it's?
Please help,
thx,
M.
--
View this message in context:
http://old.nabble.com/AMQ-%285.3.0%29-with-CMS-3.01--%3E-consumes-all-memory-and-stops-clients-and-producers-tp27771356p27771356.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: AMQ (5.3.0) with CMS-3.01 -> consumes all memory and stops clients and producers
Hi Mats,
Doesn't work neither :(
1) running AMQ version SNAPSHOT from march 03
2) config file as follows:
http://activemq.apache.org/schema/core";
brokerName="localhost" useJmx="true" dataDirectory="${activemq.base}/data">
3) this was in log when MemoryPrecentageUsage reached 100% :
2010-03-05 15:35:56,090 | INFO | Usage Manager Memory Limit reached on
temp-queue://e84351f4-d345-2d5c-44e8-35dc2523aea4:2. Producers will be
throttled to the rate at which messages are removed from this destination to
prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info |
org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///127.0.0.1:57096
again temp-queue
Maybe I should remove VMCursor or I really don't know what to do with
this ... reopen the same BUG?
Thanks,
Marcin
Mats Henrikson wrote:
>
> Hi Marcin,
>
> This looks very similar to a bug that I have logged:
>
> https://issues.apache.org/activemq/browse/AMQ-2616
>
> Supposed to be fixed in 5.3.1 and later.
>
> Mats
>
>
>
>
> On 4 March 2010 06:50, lucious wrote:
>>
>> Hello here is what we have:
>>
>> 1) AMQ -5 3.0 with the following configuration:
>> http://activemq.apache.org/schema/core";
>> brokerName="localhost" useJmx="true"
>> dataDirectory="${activemq.base}/data">
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > memoryLimit="100mb">
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > memoryLimit="100mb">
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > uri="tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=0"/>
>>
>>
>> System is being run with Xm=2048M
>>
>> 2) Producers and clients use CMS - 3.0.1 - we don't have any features
>> added
>> to CreateFactory - all defaults are used.
>>
>> Results:
>> 1) during the day watch MQ via JMX I have: totalEnqueueCount=1193470,
>> totaldequeuecount=628130,totalmessagecount=1052834 - this I think is
>> wrong
>> ... shouldn't be like that: totalmessage+totaldequeue = totalenqueue?
>>
>> 2) memorypercentusage=73 and it's counting ... once it gets to 100% all
>> is
>> frozen and in log file I get messages like this: INFO | Usage Manager
>> memory
>> limit reached on temp-queue://42455840-0bb5-0101-49b1-304ecf24dbbd:2.
>> Producers will be throttled to the rate at which messages are removed
>> from
>> this destination to prevent flooding it. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>>
>> quite strange as flow-control is disabled.
>>
>> 3) watching Heap memory Usage I can see "saw chart" but it's going up and
>> up
>> to 1GB and then is saturated (AMQ is frozen at that time).
>>
>> 4) the only thing I can do is to stop AMQ and restart ... however on
>> production is not a solution
Re: AMQ (5.3.0) with CMS-3.01 -> consumes all memory and stops clients and producers
BTW:
TotalEqueueCount: 1915k
TotalDequeueCount: 984k
TotalMessageCount: 380k
Perhaps this can lead to "solution"
M.
lucious wrote:
>
> Hi Mats,
>
> Doesn't work neither :(
>
> 1) running AMQ version SNAPSHOT from march 03
> 2) config file as follows:
> http://activemq.apache.org/schema/core";
> brokerName="localhost" useJmx="true"
> dataDirectory="${activemq.base}/data">
>
>
>
>
>
>
>
>
>
>
>
>
> memoryLimit="100mb">
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> memoryLimit="100mb" optimizedDispatch="true">
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> uri="tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=0"/>
>
>
>
>
> 3) this was in log when MemoryPrecentageUsage reached 100% :
> 2010-03-05 15:35:56,090 | INFO | Usage Manager Memory Limit reached on
> temp-queue://e84351f4-d345-2d5c-44e8-35dc2523aea4:2. Producers will be
> throttled to the rate at which messages are removed from this destination
> to prevent flooding it. See
> http://activemq.apache.org/producer-flow-control.html for more info |
> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
> tcp:///127.0.0.1:57096
>
> again temp-queue
>
>
> Maybe I should remove VMCursor or I really don't know what to do with
> this ... reopen the same BUG?
>
> Thanks,
> Marcin
>
>
>
> Mats Henrikson wrote:
>>
>> Hi Marcin,
>>
>> This looks very similar to a bug that I have logged:
>>
>> https://issues.apache.org/activemq/browse/AMQ-2616
>>
>> Supposed to be fixed in 5.3.1 and later.
>>
>> Mats
>>
>>
>>
>>
>> On 4 March 2010 06:50, lucious wrote:
>>>
>>> Hello here is what we have:
>>>
>>> 1) AMQ -5 3.0 with the following configuration:
>>> http://activemq.apache.org/schema/core";
>>> brokerName="localhost" useJmx="true"
>>> dataDirectory="${activemq.base}/data">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> memoryLimit="100mb">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> memoryLimit="100mb">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> uri="tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=0"/>
>>>
>>>
>>> System is being run with Xm=2048M
>>>
>>> 2) Producers and clients use CMS - 3.0.1 - we don't have any features
>>> added
>>> to CreateFactory - all defaults are used.
>>>
>>> Results:
>>> 1) duri
Re: AMQ (5.3.0) with CMS-3.01 -> consumes all memory and stops clients and producers
Ohh I got it ;) thanks a lot!!! BTW. What is competitors consumers? M. Hi Marcin, I think that too much messages are produced and consumers does not have enough time/memory to consume them. So the memory consumption continue to growth till to reach a limit. As you have configured the vm for the MessageCursor (http://activemq.apache.org/message-cursors.html), this behavior is logic and corresponds to what was designed in ActiveMq 4.x (see the link that I post). I propose that you removed them to let ActiveMq to use the new one which is the store front message cursor (messages are paged on disk if the memory limit is reached) or adapt the architecture to use competitors consumers. Kind regards, -- View this message in context: http://old.nabble.com/AMQ-%285.3.0%29-with-CMS-3.01--%3E-consumes-all-memory-and-stops-clients-and-producers-tp27771356p27796492.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
ActiveMQ 12.03.2010 snapshot + CMS -> memory limit reached on temp-queue and topic
Hello again,,
I tried and gave a goal with this snapshot and new configuration file -
removed vmCursors for Store/File:
http://activemq.apache.org/schema/core";
brokerName="localhost" useJmx="true" dataDirectory="${activemq.base}/data"
destroyApplicationContextOnStop="true">
but then again I faced memory limit.
2010-03-16 15:02:28,455 | INFO | Usage Manager Memory Limit reached on
temp-queue://e0499daa-604e-227c-57c5-8a700a83d690:0. Producers will be
throttled to the rate at which messages are removed from this destination to
prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info |
org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///127.0.0.1:46737
2010-03-16 15:02:28,582 | INFO | Usage Manager memory limit (536870912)
reached for temp-topic://47441d3c-6561-36a7-9904-11f5a2404d8f:0. Producers
will be throttled to the rate at which messages are removed from this
destination to prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info |
org.apache.activemq.broker.region.Topic | ActiveMQ Transport:
tcp:///127.0.0.1:44605
why is this happening? If I understand right when 70% of memory is reached
(in my case 512MB limit) all messages should go to the file? But it seems
like AMQ takes all memory until it;s exhausted and then hangs all consumers.
The only solution for this is to restart AMQ.
Maybe my configuration is still wrong, but according to information from
this forum and docs from amq I can't find any solution for this.
PLease advice,
thx
M.
--
View this message in context:
http://old.nabble.com/ActiveMQ-12.03.2010-snapshot-%2B-CMS--%3E-memory-limit-reached-on-temp-queue-and-topic-tp27924410p27924410.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
temp-queue purge from code()
Hi I filed this bug AMQ-2657 which is really annoying to me and sometimes it takes me the whole day to look at "zombie" temp-queue (no consumers but messagecount increasing) and perform purge() on such queue. Waiting for fix I'd like to ask if it's possible to create simple app to check temp-queues and if find zombie-queue then do purge() method on such queue? Any piece of code to start with? Thx, M. -- View this message in context: http://old.nabble.com/temp-queue-purge-from-code%28%29-tp28082374p28082374.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: temp-queue purge from code()
... or maybe there is a way to remove such "zombie queue" without stopping the MQ server? M. lucious wrote: > > Hi I filed this bug AMQ-2657 which is really annoying to me and sometimes > it takes me the whole day to look at "zombie" temp-queue (no consumers but > messagecount increasing) and perform purge() on such queue. Waiting for > fix I'd like to ask if it's possible to create simple app to check > temp-queues and if find zombie-queue then do purge() method on such queue? > Any piece of code to start with? > > Thx, > M. > -- View this message in context: http://old.nabble.com/temp-queue-purge-from-code%28%29-tp28082374p28083269.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: temp-queue purge from code()
Well I can spend this time and help you but I'm not that fluent with java and junit so if you can suggest how to start would be great. Thanks for this link! M. Have a look at the ruby jmx4r example at the bottom of this blog entry as a starting point on a simple way to code jmx calls : http://pavelgolubeff.ya.ru/replies.xml?item_no=1583&ncrnd=3263 It may also be worth investing some time in a junit test case for your scenario, as that would speed up finding the root cause and identifying the required fix. -- View this message in context: http://old.nabble.com/temp-queue-purge-from-code%28%29-tp28082374p28085822.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: temp-queue purge from code()
just to give a short update ... seems like there is NOW way to remove temp-queue like in this example. Normal queue can be simply removed but no temp ones. Bug or a "feature"? M. -- View this message in context: http://old.nabble.com/temp-queue-purge-from-code%28%29-tp28082374p28098574.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
