I'm testing only durable subscribers (as non-durable subscribers are working as expected as all of their messages seem being treated as non-persistent). And, I purposely set the prefetch limit to 0 (not using prefetching at all) to quickly generate the cache full of the TopicStorePrefetch class.
Please check out the test code I'm using: "prefetchLimitForAll" member variable is there to adjust prefetch limits of all types of subscriptions. MemoryLimitTest.java <http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java> ceposta wrote > Give me one more piece of information. > > What is the prefetch settings you're using for your topics (durable and > non-durable subscribers), and how many durable subscribers are connected? > > > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi < > jw@ > > wrote: > >> Thanks a lot Christian for the well-explained documentation and this >> explains >> why I'm seeing the trouble with topic durable subscriptions regarding the >> per-destination memory usage & limit. >> >> First, let me quote some from your writing: >> >> "Main Broker Memory, Destination Memory, Subscription Memory >> >> ... >> >> A destination, when it’s created, will create its own SystemUsage object >> (which creates its own separate Memory, Store, and Temp Usage objects) >> but >> it will set its parent to the be broker’s main SystemUsage object. A >> destination can have its memory limits tuned individually (but not Store >> and >> Temp, those will still delegate to the parent). To set a destination’s >> memory limit: >> ... >> >> So the destination usage objects can be used to more finely control >> MemoryUsage, but it will always coordinate with the Main memory for all >> usage counts. This functionality can be used to limit the number of >> messages >> that a destination keeps around so that a single destination cannot >> starve >> other destinations. *For queues, it also affects the store cursor’s high >> water mark. A queue has different cursors for persistent and >> non-persistent >> messages. If we hit the high water mark (a threshold of the destination’s >> memory limit), no more messages be cached ready to be dispatched, and >> non-persistent messages can be purged to temp disk as necessary (if the >> StoreCursor will use FilePendingMessageCursor… otherwise it will just use >> a >> VMPendingMessageCursor and won’t purge to temporary store).* >> >> If you don’t specify a memory limit for individual destinations, the >> destination’s SystemUsage will delegate to the parent (Main SystemUsage) >> for >> all usage counts. This means it will effectively use the broker’s Main >> SystemUsage for all memory-related counts. >> >> *Consumer subscriptions, on the other hand, don’t have any notion of >> their >> own SystemUsage or MemoryUsage counters. They will always use the >> broker’s >> Main SystemUsage objects.* The main thing to note about this is when >> using >> a >> FilePendingMessageCursor for subscriptions (for example, for a Topic >> subscription), the messages will not be swapped to disk until the cursor >> high-water mark (70% by default) is reached.. but that means 70% of Main >> memory will need to be reached. That could be a while, and a lot of >> messages >> could be kept in memory. And if your subscription is the one holding most >> of >> those messages, swapping to disk could take a while. As topics dispatch >> messages to one subscription at a time, if one subscription grinds to a >> halt >> because it’s swapping its messages to disk, the rest of the subscription >> ready to receive the message will also feel the slow down..." >> >> What I don't understand from the above is why consumer subscriptions (and >> their cursors) of topics are not using per-destination MemoryLimit but >> share >> the Broker's main MemoryLimit, unlike the queues (please see the bolded >> sentences above). Due to this + some logics inside AbstractStoreCursor's >> space checking logics, PFC always kicks in for the topic whenever the >> persistent store cursor's cache (pendingList) gets full. Let me explain >> what's happening: >> >> 1. A topic is created, with per-destination memory limit = 1MB and >> broker's >> main memory limit = 5MB. >> 2. Topic's SystemUsage (for PFC purpose) is configured with >> per-destination >> memory limit (1MB), while TopicStorePrefetch(persistent store cursor)'s >> SystemUsage (for remaining cache space checking purpose) is configured >> with >> broker's main memory limit (10MB). >> 3. The 1st message of 0.8MB is published: >> a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull is >> true. >> b. It's cached to the persistent cursor, as >> AbstractStoreCursor#hasSpace >> is true. >> c. per-dest memory usage % becomes 90%, while main memory usage % >> becomes >> 9%. >> 4. The 2nd message of 0.8MB is published: >> a. PFC doesn't kick in either, as per-dest memory usage is 90% (< >> 100%). >> This is fine. >> b. *(I expect here that this second message shouldn't be cached to the >> cursor but invalidate it, but)this second message is also cached to the >> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% < >> 70% >> (main memory usage high watermark)!!!* >> c. per-dest memory usage % becomes 180%, while main memory usage % >> becomes 18%. >> 5. The 3rd message of 0.8MB is published: >> a. *(I don't want this behavior but) PFC kicks in!!!* because >> per-destination memory limit is exceeded (180%). >> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks >> cursor >> memory availability against per-dest memory limit (not against broker's >> main >> memory limit) so that cursor pending cache gets disabled (just keeps what >> has been cached) w/o increasing the per-destination memory usage. >> >> I don't want to block persistent message publishing to topics by the >> per-destination memory limit but only for non-persistent message >> publishing, >> cause the store size allowed for a destination is relatively higher than >> the >> memory limit: e.g. I'd like to allow per-destination persistent message >> publishing up to 10GB disk space without being blocked by the memory >> limit >> & >> PFC... How can I achieve this? >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta -- View this message in context: http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
