Inline....
On Thu, Jun 6, 2013 at 8:08 AM, Jesus Roncero <je...@we7.com> wrote: > On Wed, Jun 05, 2013 at 08:57:01AM -0700, Christian Posta wrote: > > Hi, > > further on this: > > > Sounds like it's happening exactly as you have it configured.. I assume > > you're using non-persistent messages? > > > > As Johan pointed out, your config kinda doesn't make sense :) > > > > The <memoryUsage> is an overall broker limit. That is, if you set your > > <memoryUsage limit="40mb"> then you want to keep the aggregate of all of > > your destinations <= 40mb. Therefore, if you have 40 queues, set each one > > to 1mb. Or some other combo that is <= 40mb. In your case you set each > > queue to 50mb (<policyEntry memoryLimit="50mb"...) which doesn't work > well. > > As for your questions... > > > > 1) the 50mb limit doesn't kick in (you don't get PFC) because the broker > > has hit the 70% limit and is spooling your messages to disk. in the first > > case, it never hits the 70% mem limit of your destination because it > filled > > up (40mb) the broker main memory first > > Agreed, but if I have a config more like scenario 2. > > > 2) it sure is happening :) check the logs for scenario #2... you hit the > > limit on the temp store > > Sure as I understand then, the 70% kicks in and passes it over to the temp > storage, until this fills? > Well... when 70% kicks in, it will store to temp storage. If it's over the temp limit, it still stores it. > > This is the config I'm testing: > > <policyEntry queue=">" memoryLimit="50mb" > producerFlowControl="false"> > <pendingQueuePolicy> > <fileQueueCursor /> > </pendingQueuePolicy> > </policyEntry> > <policyEntry topic=">" memoryLimit="50mb" > producerFlowControl="false"> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="300 mb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb" /> > </storeUsage> > <tempUsage> > <tempUsage limit="800 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > and the logs I'm seeing are: > > [2013-06-06 15:37:23,283] INFO BrokerService.worker.1 > org.apache.activemq.store.kahadb.plist.PListStore - > PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized > 2013-06-06 15:37:23,283 [ervice.worker.1] INFO PListStore - > PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized > [2013-06-06 15:37:33,997] INFO ActiveMQ Transport: > tcp:///10.0.9.124:46903org.apache.activemq.broker.region.Queue - > Usage(default:temp:queue://testqueue:temp) percentUsage=99%, > usage=839122944, limit=838860800, > percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, > usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is > Full (99% of 838860800). Stopping producer > (ID:meme-52453-1370529255477-2:1:-1:1) to prevent flooding > queue://testqueue. See > http://activemq.apache.org/producer-flow-control.html for more info > (blocking for: 1s) > 2013-06-06 15:37:33,997 [0.0.9.124:46903] INFO Queue - > Usage(default:temp:queue://testqueue:temp) percentUsage=99%, > usage=839122944, limit=838860800, > percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, > usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is > Full (99% of 838860800). Stopping producer > (ID:meme-52453-1370529255477-2:1:-1:1) to prevent flooding > queue://testqueue. See > http://activemq.apache.org/producer-flow-control.html for more info > (blocking for: 1s) > > > So, here, having now flow control, the messages are stored into the > temporary > Area, until it fills up. But if I change the setting to have > producerFlowControl, > then the behaviour is exactly the same: > > [2013-06-06 15:56:32,576] INFO BrokerService.worker.1 > org.apache.activemq.store.kahadb.plist.PListStore - > PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized > 2013-06-06 15:56:32,576 [ervice.worker.1] INFO PListStore - > PListStore:[/var/lib/activemq/we7/data/localhost/tmp_storage ] initialized > [2013-06-06 15:56:43,426] INFO ActiveMQ Transport: > tcp:///10.0.9.124:47262org.apache.activemq.broker.region.Queue - > Usage(default:temp:queue://testqueue:temp) percentUsage=99%, > usage=839122944, limit=838860800, > percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, > usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is > Full (99% of 838860800). Stopping producer > (ID:meme-59314-1370530472794-2:1:-1:1) to prevent flooding > queue://testqueue. See > http://activemq.apache.org/producer-flow-control.html for more info > (blocking for: 1s) > 2013-06-06 15:56:43,426 [0.0.9.124:47262] INFO Queue - > Usage(default:temp:queue://testqueue:temp) percentUsage=99%, > usage=839122944, limit=838860800, > percentUsageMinDelta=1%;Parent:Usage(default:temp) percentUsage=100%, > usage=839122944, limit=838860800, percentUsageMinDelta=1%: Temp Store is > Full (99% of 838860800). Stopping producer > (ID:meme-59314-1370530472794-2:1:-1:1) to prevent flooding > queue://testqueue. See > http://activemq.apache.org/producer-flow-control.html for more info > (blocking for: 1s) > > So, > > 1) how these two different configurations differ, if I'm seeing the same > behaviour? Shouldn't it have stopped based on the memory limit of each > of the queue? > When you have producer flow control enabled, your producers should block when you've reached the memory limits (50mb). If you can write a unit test that shows otherwise, I would gladly take a look. > > 2) I'm assuming that this is the right behaviour for activeMQ? I mean, > assuming I have sensible memory limits and temp storage, this is how it's > supposed to behave? > > 3) Regardless, I should be informing the consumers that the temp storage is > full rather than making them hang? This setting (sendFailIfNoSpace) seems > not > to make any difference whether producer flow control is enabled or not. > Again, if you can write a test, we can see what's going on. When sendFailIfNoSpace is set to true, it should send back a ResourceAllocationException even for temp storage. > > 4) What's the relationship between memoryUage (global config option) and > the > java heap space defined by -Xms512M in the config? > Think of memoryUsage as a slice or chunk of the JVM heap. We use it to keep track of how many messages we're keeping around in memory, so we don't OOM. However, if producer flow control is off, then these limits aren't checked, and when using non-persistent messages, you have to take other measures to make sure you don't OOM. > Again, if you could shed some light and help me undrstand this, it'd be > hightly appreciated. Thanks. > > Regards. > -- > JRF > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
