On Wed, Jun 05, 2013 at 08:57:01AM -0700, Christian Posta wrote:
Hi,
further on this:
> Sounds like it's happening exactly as you have it configured.. I assume
> you're using non-persistent messages?
>
> As Johan pointed out, your config kinda doesn't make sense :)
>
> The <memoryUsage> is an overall broker limit. That is, if you set your
> <memoryUsage limit="40mb"> then you want to keep the aggregate of all of
> your destinations <= 40mb. Therefore, if you have 40 queues, set each one
> to 1mb. Or some other combo that is <= 40mb. In your case you set each
> queue to 50mb (<policyEntry memoryLimit="50mb"...) which doesn't work well.
> As for your questions...
>
> 1) the 50mb limit doesn't kick in (you don't get PFC) because the broker
> has hit the 70% limit and is spooling your messages to disk. in the first
> case, it never hits the 70% mem limit of your destination because it filled
> up (40mb) the broker main memory first
Agreed, but if I have a config more like scenario 2.
> 2) it sure is happening :) check the logs for scenario #2... you hit the
> limit on the temp store
Sure as I understand then, the 70% kicks in and passes it over to the temp
storage, until this fills?
This is the config I'm testing:
<policyEntry queue=">" memoryLimit="50mb" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor />
</pendingQueuePolicy>
</policyEntry>
<policyEntry topic=">" memoryLimit="50mb" producerFlowControl="false">
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="300 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="800 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
and the logs I'm seeing are:
[2013-06-06 15:37:23,283] INFO BrokerService.worker.1
org.apache.activemq.store.kahadb.plist.PListStore -
PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
2013-06-06 15:37:23,283 [ervice.worker.1] INFO PListStore -
PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
[2013-06-06 15:37:33,997] INFO ActiveMQ Transport: tcp:///10.0.9.124:46903
org.apache.activemq.broker.region.Queue -
Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944,
limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp)
percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%:
Temp Store is Full (99% of 838860800). Stopping producer
(ID:meme-52453-1370529255477-2:1:-1:1) to prevent flooding queue://testqueue.
See http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 1s)
2013-06-06 15:37:33,997 [0.0.9.124:46903] INFO Queue -
Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944,
limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp)
percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%:
Temp Store is Full (99% of 838860800). Stopping producer
(ID:meme-52453-1370529255477-2:1:-1:1) to prevent flooding queue://testqueue.
See http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 1s)
So, here, having now flow control, the messages are stored into the temporary
Area, until it fills up. But if I change the setting to have
producerFlowControl,
then the behaviour is exactly the same:
[2013-06-06 15:56:32,576] INFO BrokerService.worker.1
org.apache.activemq.store.kahadb.plist.PListStore -
PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
2013-06-06 15:56:32,576 [ervice.worker.1] INFO PListStore -
PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized
[2013-06-06 15:56:43,426] INFO ActiveMQ Transport: tcp:///10.0.9.124:47262
org.apache.activemq.broker.region.Queue -
Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944,
limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp)
percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%:
Temp Store is Full (99% of 838860800). Stopping producer
(ID:meme-59314-1370530472794-2:1:-1:1) to prevent flooding queue://testqueue.
See http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 1s)
2013-06-06 15:56:43,426 [0.0.9.124:47262] INFO Queue -
Usage(default:temp:queue://testqueue:temp) percentUsage=99%, usage=839122944,
limit=838860800, percentUsageMinDelta=1%;Parent:Usage(default:temp)
percentUsage=100%, usage=839122944, limit=838860800, percentUsageMinDelta=1%:
Temp Store is Full (99% of 838860800). Stopping producer
(ID:meme-59314-1370530472794-2:1:-1:1) to prevent flooding queue://testqueue.
See http://activemq.apache.org/producer-flow-control.html for more info
(blocking for: 1s)
So,
1) how these two different configurations differ, if I'm seeing the same
behaviour? Shouldn't it have stopped based on the memory limit of each
of the queue?
2) I'm assuming that this is the right behaviour for activeMQ? I mean,
assuming I have sensible memory limits and temp storage, this is how it's
supposed to behave?
3) Regardless, I should be informing the consumers that the temp storage is
full rather than making them hang? This setting (sendFailIfNoSpace) seems not
to make any difference whether producer flow control is enabled or not.
4) What's the relationship between memoryUage (global config option) and the
java heap space defined by -Xms512M in the config?
Again, if you could shed some light and help me undrstand this, it'd be
hightly appreciated. Thanks.
Regards.
--
JRF
