Give me one more piece of information. What is the prefetch settings you're using for your topics (durable and non-durable subscribers), and how many durable subscribers are connected?
On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi <j...@yahoo-inc.com> wrote: > Thanks a lot Christian for the well-explained documentation and this > explains > why I'm seeing the trouble with topic durable subscriptions regarding the > per-destination memory usage & limit. > > First, let me quote some from your writing: > > "Main Broker Memory, Destination Memory, Subscription Memory > > ... > > A destination, when it’s created, will create its own SystemUsage object > (which creates its own separate Memory, Store, and Temp Usage objects) but > it will set its parent to the be broker’s main SystemUsage object. A > destination can have its memory limits tuned individually (but not Store > and > Temp, those will still delegate to the parent). To set a destination’s > memory limit: > ... > > So the destination usage objects can be used to more finely control > MemoryUsage, but it will always coordinate with the Main memory for all > usage counts. This functionality can be used to limit the number of > messages > that a destination keeps around so that a single destination cannot starve > other destinations. *For queues, it also affects the store cursor’s high > water mark. A queue has different cursors for persistent and non-persistent > messages. If we hit the high water mark (a threshold of the destination’s > memory limit), no more messages be cached ready to be dispatched, and > non-persistent messages can be purged to temp disk as necessary (if the > StoreCursor will use FilePendingMessageCursor… otherwise it will just use a > VMPendingMessageCursor and won’t purge to temporary store).* > > If you don’t specify a memory limit for individual destinations, the > destination’s SystemUsage will delegate to the parent (Main SystemUsage) > for > all usage counts. This means it will effectively use the broker’s Main > SystemUsage for all memory-related counts. > > *Consumer subscriptions, on the other hand, don’t have any notion of their > own SystemUsage or MemoryUsage counters. They will always use the broker’s > Main SystemUsage objects.* The main thing to note about this is when using > a > FilePendingMessageCursor for subscriptions (for example, for a Topic > subscription), the messages will not be swapped to disk until the cursor > high-water mark (70% by default) is reached.. but that means 70% of Main > memory will need to be reached. That could be a while, and a lot of > messages > could be kept in memory. And if your subscription is the one holding most > of > those messages, swapping to disk could take a while. As topics dispatch > messages to one subscription at a time, if one subscription grinds to a > halt > because it’s swapping its messages to disk, the rest of the subscription > ready to receive the message will also feel the slow down..." > > What I don't understand from the above is why consumer subscriptions (and > their cursors) of topics are not using per-destination MemoryLimit but > share > the Broker's main MemoryLimit, unlike the queues (please see the bolded > sentences above). Due to this + some logics inside AbstractStoreCursor's > space checking logics, PFC always kicks in for the topic whenever the > persistent store cursor's cache (pendingList) gets full. Let me explain > what's happening: > > 1. A topic is created, with per-destination memory limit = 1MB and broker's > main memory limit = 5MB. > 2. Topic's SystemUsage (for PFC purpose) is configured with per-destination > memory limit (1MB), while TopicStorePrefetch(persistent store cursor)'s > SystemUsage (for remaining cache space checking purpose) is configured with > broker's main memory limit (10MB). > 3. The 1st message of 0.8MB is published: > a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull is > true. > b. It's cached to the persistent cursor, as AbstractStoreCursor#hasSpace > is true. > c. per-dest memory usage % becomes 90%, while main memory usage % > becomes > 9%. > 4. The 2nd message of 0.8MB is published: > a. PFC doesn't kick in either, as per-dest memory usage is 90% (< 100%). > This is fine. > b. *(I expect here that this second message shouldn't be cached to the > cursor but invalidate it, but)this second message is also cached to the > persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% < 70% > (main memory usage high watermark)!!!* > c. per-dest memory usage % becomes 180%, while main memory usage % > becomes 18%. > 5. The 3rd message of 0.8MB is published: > a. *(I don't want this behavior but) PFC kicks in!!!* because > per-destination memory limit is exceeded (180%). > > The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks cursor > memory availability against per-dest memory limit (not against broker's > main > memory limit) so that cursor pending cache gets disabled (just keeps what > has been cached) w/o increasing the per-destination memory usage. > > I don't want to block persistent message publishing to topics by the > per-destination memory limit but only for non-persistent message > publishing, > cause the store size allowed for a destination is relatively higher than > the > memory limit: e.g. I'd like to allow per-destination persistent message > publishing up to 10GB disk space without being blocked by the memory limit > & > PFC... How can I achieve this? > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
