@Christian: Sorry for bugging you bug wonder if you've gotten a chance to test the queue case as well, as I suggested. Or please shed the light once more if I'm still missing or misunderstanding something.
Regards, Jake Jake Choi wrote > Thanks for your test. But I still don't understand your point, meaning > what you explained with prefetch-limit doesn't make sense to me. > > As I understand, prefetch limit has no relation with PFC as it's there to > throttle prefetching to consumers and possibly block message consumption > from the consumer who has a lot messages left unacknowledged: > http://activemq.apache.org/what-is-the-prefetch-limit-for.html > > I understand PFC has relation with the backlog size (cached pending list > size) but I think it should do only for non-persistent message publishing. > Why do we need to block persistent message publishing by memory limit? > Persistent messages could be contributing to the memory usage but until > the cache limit is reached with the high-watermark (70%). After that, > broker can disable the cache/stopping caching(cursor dispatching) the > persistent message but keep writing only to the backed persistent store > w/o increasing the cache size (memory usage) anymore. This is not a new > but the way how Queue works for persistent message publishing for now, as > you documented: / >> For queues, it also affects the store cursor’s high water mark. A queue >> has different cursors for persistent and non-persistent messages. If we >> hit the high water mark (a threshold of the destination’s memory limit), >> no more messages be cached ready to be dispatched, and non-persistent >> messages can be purged to temp disk as necessary (if the StoreCursor will >> use FilePendingMessageCursor… otherwise it will just use a >> VMPendingMessageCursor and won’t purge to temporary store). / > > Could you please kindly do the same test with the Queue? (please find > "testPersistentQueue" method) e.g. prefetchLimit=0 and publish persistent > messages to the Queue which is configured with Store-based cursor for the > back-ed storage. Regardless of the prefetchLimit setting on the queue > consumer side, you'll be able to publish as many as messages to the > back-ed storage not being blocked by any memory limit until it hits the > storage limit.. > ceposta wrote >> Thanks for the test... >> I played around with it for a bit, and everything seems to work as I >> expected. >> >> Memory usage for durable subs on topics only kicks in when dispatch to >> the >> consumer has already hit the prefetch limit. That means, try altering >> your >> prefetch limits, and you'll notice that the number of messages that can >> be >> dispatched to a live durable consumer without hitting PFC changes along >> with the consumer prefetch. Once you've hit the prefetch limit, all of >> those messages get cached and will count against memory usage and >> eventually you'll hit PFC. >> >> Does that make sense? >> >> >> On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi < >> jw@ >> > wrote: >> >>> I'm testing only durable subscribers (as non-durable subscribers are >>> working >>> as expected as all of their messages seem being treated as >>> non-persistent). >>> And, I purposely set the prefetch limit to 0 (not using prefetching at >>> all) >>> to quickly generate the cache full of the TopicStorePrefetch class. >>> >>> Please check out the test code I'm using: "prefetchLimitForAll" member >>> variable is there to adjust prefetch limits of all types of >>> subscriptions. >>> >>> MemoryLimitTest.java >>> <http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java> >>> >>> >>> ceposta wrote >>> > Give me one more piece of information. >>> > >>> > What is the prefetch settings you're using for your topics (durable >>> and >>> > non-durable subscribers), and how many durable subscribers are >>> connected? >>> > >>> > >>> > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi < >>> >>> > jw@ >>> >>> > > wrote: >>> > >>> >> Thanks a lot Christian for the well-explained documentation and this >>> >> explains >>> >> why I'm seeing the trouble with topic durable subscriptions regarding >>> the >>> >> per-destination memory usage & limit. >>> >> >>> >> First, let me quote some from your writing: >>> >> >>> >> "Main Broker Memory, Destination Memory, Subscription Memory >>> >> >>> >> ... >>> >> >>> >> A destination, when it’s created, will create its own SystemUsage >>> object >>> >> (which creates its own separate Memory, Store, and Temp Usage >>> objects) >>> >> but >>> >> it will set its parent to the be broker’s main SystemUsage object. A >>> >> destination can have its memory limits tuned individually (but not >>> Store >>> >> and >>> >> Temp, those will still delegate to the parent). To set a >>> destination’s >>> >> memory limit: >>> >> ... >>> >> >>> >> So the destination usage objects can be used to more finely control >>> >> MemoryUsage, but it will always coordinate with the Main memory for >>> all >>> >> usage counts. This functionality can be used to limit the number of >>> >> messages >>> >> that a destination keeps around so that a single destination cannot >>> >> starve >>> >> other destinations. *For queues, it also affects the store cursor’s >>> high >>> >> water mark. A queue has different cursors for persistent and >>> >> non-persistent >>> >> messages. If we hit the high water mark (a threshold of the >>> destination’s >>> >> memory limit), no more messages be cached ready to be dispatched, and >>> >> non-persistent messages can be purged to temp disk as necessary (if >>> the >>> >> StoreCursor will use FilePendingMessageCursor… otherwise it will just >>> use >>> >> a >>> >> VMPendingMessageCursor and won’t purge to temporary store).* >>> >> >>> >> If you don’t specify a memory limit for individual destinations, the >>> >> destination’s SystemUsage will delegate to the parent (Main >>> SystemUsage) >>> >> for >>> >> all usage counts. This means it will effectively use the broker’s >>> Main >>> >> SystemUsage for all memory-related counts. >>> >> >>> >> *Consumer subscriptions, on the other hand, don’t have any notion of >>> >> their >>> >> own SystemUsage or MemoryUsage counters. They will always use the >>> >> broker’s >>> >> Main SystemUsage objects.* The main thing to note about this is when >>> >> using >>> >> a >>> >> FilePendingMessageCursor for subscriptions (for example, for a Topic >>> >> subscription), the messages will not be swapped to disk until the >>> cursor >>> >> high-water mark (70% by default) is reached.. but that means 70% of >>> Main >>> >> memory will need to be reached. That could be a while, and a lot of >>> >> messages >>> >> could be kept in memory. And if your subscription is the one holding >>> most >>> >> of >>> >> those messages, swapping to disk could take a while. As topics >>> dispatch >>> >> messages to one subscription at a time, if one subscription grinds to >>> a >>> >> halt >>> >> because it’s swapping its messages to disk, the rest of the >>> subscription >>> >> ready to receive the message will also feel the slow down..." >>> >> >>> >> What I don't understand from the above is why consumer subscriptions >>> (and >>> >> their cursors) of topics are not using per-destination MemoryLimit >>> but >>> >> share >>> >> the Broker's main MemoryLimit, unlike the queues (please see the >>> bolded >>> >> sentences above). Due to this + some logics inside >>> AbstractStoreCursor's >>> >> space checking logics, PFC always kicks in for the topic whenever the >>> >> persistent store cursor's cache (pendingList) gets full. Let me >>> explain >>> >> what's happening: >>> >> >>> >> 1. A topic is created, with per-destination memory limit = 1MB and >>> >> broker's >>> >> main memory limit = 5MB. >>> >> 2. Topic's SystemUsage (for PFC purpose) is configured with >>> >> per-destination >>> >> memory limit (1MB), while TopicStorePrefetch(persistent store >>> cursor)'s >>> >> SystemUsage (for remaining cache space checking purpose) is >>> configured >>> >> with >>> >> broker's main memory limit (10MB). >>> >> 3. The 1st message of 0.8MB is published: >>> >> a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull >>> is >>> >> true. >>> >> b. It's cached to the persistent cursor, as >>> >> AbstractStoreCursor#hasSpace >>> >> is true. >>> >> c. per-dest memory usage % becomes 90%, while main memory usage % >>> >> becomes >>> >> 9%. >>> >> 4. The 2nd message of 0.8MB is published: >>> >> a. PFC doesn't kick in either, as per-dest memory usage is 90% (< >>> >> 100%). >>> >> This is fine. >>> >> b. *(I expect here that this second message shouldn't be cached to >>> the >>> >> cursor but invalidate it, but)this second message is also cached to >>> the >>> >> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% >>> < >>> >> 70% >>> >> (main memory usage high watermark)!!!* >>> >> c. per-dest memory usage % becomes 180%, while main memory usage % >>> >> becomes 18%. >>> >> 5. The 3rd message of 0.8MB is published: >>> >> a. *(I don't want this behavior but) PFC kicks in!!!* because >>> >> per-destination memory limit is exceeded (180%). >>> >> >>> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks >>> >> cursor >>> >> memory availability against per-dest memory limit (not against >>> broker's >>> >> main >>> >> memory limit) so that cursor pending cache gets disabled (just keeps >>> what >>> >> has been cached) w/o increasing the per-destination memory usage. >>> >> >>> >> I don't want to block persistent message publishing to topics by the >>> >> per-destination memory limit but only for non-persistent message >>> >> publishing, >>> >> cause the store size allowed for a destination is relatively higher >>> than >>> >> the >>> >> memory limit: e.g. I'd like to allow per-destination persistent >>> message >>> >> publishing up to 10GB disk space without being blocked by the memory >>> >> limit >>> >> & >>> >> PFC... How can I achieve this? >>> >> >>> >> >>> >> >>> >> -- >>> >> View this message in context: >>> >> >>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >> >>> > >>> > >>> > >>> > -- >>> > *Christian Posta* >>> > http://www.christianposta.com/blog >>> > twitter: @christianposta >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >> >> >> >> -- >> *Christian Posta* >> http://www.christianposta.com/blog >> twitter: @christianposta -- View this message in context: http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4670001.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
