Thanks for the test... I played around with it for a bit, and everything seems to work as I expected.
Memory usage for durable subs on topics only kicks in when dispatch to the consumer has already hit the prefetch limit. That means, try altering your prefetch limits, and you'll notice that the number of messages that can be dispatched to a live durable consumer without hitting PFC changes along with the consumer prefetch. Once you've hit the prefetch limit, all of those messages get cached and will count against memory usage and eventually you'll hit PFC. Does that make sense? On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi <j...@yahoo-inc.com> wrote: > I'm testing only durable subscribers (as non-durable subscribers are > working > as expected as all of their messages seem being treated as non-persistent). > And, I purposely set the prefetch limit to 0 (not using prefetching at all) > to quickly generate the cache full of the TopicStorePrefetch class. > > Please check out the test code I'm using: "prefetchLimitForAll" member > variable is there to adjust prefetch limits of all types of subscriptions. > > MemoryLimitTest.java > <http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java> > > > ceposta wrote > > Give me one more piece of information. > > > > What is the prefetch settings you're using for your topics (durable and > > non-durable subscribers), and how many durable subscribers are connected? > > > > > > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi < > > > jw@ > > > > wrote: > > > >> Thanks a lot Christian for the well-explained documentation and this > >> explains > >> why I'm seeing the trouble with topic durable subscriptions regarding > the > >> per-destination memory usage & limit. > >> > >> First, let me quote some from your writing: > >> > >> "Main Broker Memory, Destination Memory, Subscription Memory > >> > >> ... > >> > >> A destination, when it’s created, will create its own SystemUsage object > >> (which creates its own separate Memory, Store, and Temp Usage objects) > >> but > >> it will set its parent to the be broker’s main SystemUsage object. A > >> destination can have its memory limits tuned individually (but not Store > >> and > >> Temp, those will still delegate to the parent). To set a destination’s > >> memory limit: > >> ... > >> > >> So the destination usage objects can be used to more finely control > >> MemoryUsage, but it will always coordinate with the Main memory for all > >> usage counts. This functionality can be used to limit the number of > >> messages > >> that a destination keeps around so that a single destination cannot > >> starve > >> other destinations. *For queues, it also affects the store cursor’s high > >> water mark. A queue has different cursors for persistent and > >> non-persistent > >> messages. If we hit the high water mark (a threshold of the > destination’s > >> memory limit), no more messages be cached ready to be dispatched, and > >> non-persistent messages can be purged to temp disk as necessary (if the > >> StoreCursor will use FilePendingMessageCursor… otherwise it will just > use > >> a > >> VMPendingMessageCursor and won’t purge to temporary store).* > >> > >> If you don’t specify a memory limit for individual destinations, the > >> destination’s SystemUsage will delegate to the parent (Main SystemUsage) > >> for > >> all usage counts. This means it will effectively use the broker’s Main > >> SystemUsage for all memory-related counts. > >> > >> *Consumer subscriptions, on the other hand, don’t have any notion of > >> their > >> own SystemUsage or MemoryUsage counters. They will always use the > >> broker’s > >> Main SystemUsage objects.* The main thing to note about this is when > >> using > >> a > >> FilePendingMessageCursor for subscriptions (for example, for a Topic > >> subscription), the messages will not be swapped to disk until the cursor > >> high-water mark (70% by default) is reached.. but that means 70% of Main > >> memory will need to be reached. That could be a while, and a lot of > >> messages > >> could be kept in memory. And if your subscription is the one holding > most > >> of > >> those messages, swapping to disk could take a while. As topics dispatch > >> messages to one subscription at a time, if one subscription grinds to a > >> halt > >> because it’s swapping its messages to disk, the rest of the subscription > >> ready to receive the message will also feel the slow down..." > >> > >> What I don't understand from the above is why consumer subscriptions > (and > >> their cursors) of topics are not using per-destination MemoryLimit but > >> share > >> the Broker's main MemoryLimit, unlike the queues (please see the bolded > >> sentences above). Due to this + some logics inside > AbstractStoreCursor's > >> space checking logics, PFC always kicks in for the topic whenever the > >> persistent store cursor's cache (pendingList) gets full. Let me explain > >> what's happening: > >> > >> 1. A topic is created, with per-destination memory limit = 1MB and > >> broker's > >> main memory limit = 5MB. > >> 2. Topic's SystemUsage (for PFC purpose) is configured with > >> per-destination > >> memory limit (1MB), while TopicStorePrefetch(persistent store cursor)'s > >> SystemUsage (for remaining cache space checking purpose) is configured > >> with > >> broker's main memory limit (10MB). > >> 3. The 1st message of 0.8MB is published: > >> a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull is > >> true. > >> b. It's cached to the persistent cursor, as > >> AbstractStoreCursor#hasSpace > >> is true. > >> c. per-dest memory usage % becomes 90%, while main memory usage % > >> becomes > >> 9%. > >> 4. The 2nd message of 0.8MB is published: > >> a. PFC doesn't kick in either, as per-dest memory usage is 90% (< > >> 100%). > >> This is fine. > >> b. *(I expect here that this second message shouldn't be cached to > the > >> cursor but invalidate it, but)this second message is also cached to the > >> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% < > >> 70% > >> (main memory usage high watermark)!!!* > >> c. per-dest memory usage % becomes 180%, while main memory usage % > >> becomes 18%. > >> 5. The 3rd message of 0.8MB is published: > >> a. *(I don't want this behavior but) PFC kicks in!!!* because > >> per-destination memory limit is exceeded (180%). > >> > >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks > >> cursor > >> memory availability against per-dest memory limit (not against broker's > >> main > >> memory limit) so that cursor pending cache gets disabled (just keeps > what > >> has been cached) w/o increasing the per-destination memory usage. > >> > >> I don't want to block persistent message publishing to topics by the > >> per-destination memory limit but only for non-persistent message > >> publishing, > >> cause the store size allowed for a destination is relatively higher than > >> the > >> memory limit: e.g. I'd like to allow per-destination persistent message > >> publishing up to 10GB disk space without being blocked by the memory > >> limit > >> & > >> PFC... How can I achieve this? > >> > >> > >> > >> -- > >> View this message in context: > >> > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > > > > > > > > -- > > *Christian Posta* > > http://www.christianposta.com/blog > > twitter: @christianposta > > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
