actually looking through it right now :)
On Thu, Aug 1, 2013 at 9:47 AM, Jake Choi <j...@yahoo-inc.com> wrote: > @Christian: Sorry for bugging you bug wonder if you've gotten a chance to > test the queue case as well, as I suggested. Or please shed the light once > more if I'm still missing or misunderstanding something. > > Regards, > Jake > > > Jake Choi wrote > > Thanks for your test. But I still don't understand your point, meaning > > what you explained with prefetch-limit doesn't make sense to me. > > > > As I understand, prefetch limit has no relation with PFC as it's there to > > throttle prefetching to consumers and possibly block message consumption > > from the consumer who has a lot messages left unacknowledged: > > http://activemq.apache.org/what-is-the-prefetch-limit-for.html > > > > I understand PFC has relation with the backlog size (cached pending list > > size) but I think it should do only for non-persistent message > publishing. > > Why do we need to block persistent message publishing by memory limit? > > Persistent messages could be contributing to the memory usage but until > > the cache limit is reached with the high-watermark (70%). After that, > > broker can disable the cache/stopping caching(cursor dispatching) the > > persistent message but keep writing only to the backed persistent store > > w/o increasing the cache size (memory usage) anymore. This is not a new > > but the way how Queue works for persistent message publishing for now, as > > you documented: > / > >> For queues, it also affects the store cursor’s high water mark. A queue > >> has different cursors for persistent and non-persistent messages. If we > >> hit the high water mark (a threshold of the destination’s memory limit), > >> no more messages be cached ready to be dispatched, and non-persistent > >> messages can be purged to temp disk as necessary (if the StoreCursor > will > >> use FilePendingMessageCursor… otherwise it will just use a > >> VMPendingMessageCursor and won’t purge to temporary store). > / > > > > Could you please kindly do the same test with the Queue? (please find > > "testPersistentQueue" method) e.g. prefetchLimit=0 and publish persistent > > messages to the Queue which is configured with Store-based cursor for the > > back-ed storage. Regardless of the prefetchLimit setting on the queue > > consumer side, you'll be able to publish as many as messages to the > > back-ed storage not being blocked by any memory limit until it hits the > > storage limit.. > > ceposta wrote > >> Thanks for the test... > >> I played around with it for a bit, and everything seems to work as I > >> expected. > >> > >> Memory usage for durable subs on topics only kicks in when dispatch to > >> the > >> consumer has already hit the prefetch limit. That means, try altering > >> your > >> prefetch limits, and you'll notice that the number of messages that can > >> be > >> dispatched to a live durable consumer without hitting PFC changes along > >> with the consumer prefetch. Once you've hit the prefetch limit, all of > >> those messages get cached and will count against memory usage and > >> eventually you'll hit PFC. > >> > >> Does that make sense? > >> > >> > >> On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi < > > >> jw@ > > >> > wrote: > >> > >>> I'm testing only durable subscribers (as non-durable subscribers are > >>> working > >>> as expected as all of their messages seem being treated as > >>> non-persistent). > >>> And, I purposely set the prefetch limit to 0 (not using prefetching at > >>> all) > >>> to quickly generate the cache full of the TopicStorePrefetch class. > >>> > >>> Please check out the test code I'm using: "prefetchLimitForAll" member > >>> variable is there to adjust prefetch limits of all types of > >>> subscriptions. > >>> > >>> MemoryLimitTest.java > >>> < > http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java> > ; > >>> > >>> > >>> ceposta wrote > >>> > Give me one more piece of information. > >>> > > >>> > What is the prefetch settings you're using for your topics (durable > >>> and > >>> > non-durable subscribers), and how many durable subscribers are > >>> connected? > >>> > > >>> > > >>> > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi < > >>> > >>> > jw@ > >>> > >>> > > wrote: > >>> > > >>> >> Thanks a lot Christian for the well-explained documentation and this > >>> >> explains > >>> >> why I'm seeing the trouble with topic durable subscriptions > regarding > >>> the > >>> >> per-destination memory usage & limit. > >>> >> > >>> >> First, let me quote some from your writing: > >>> >> > >>> >> "Main Broker Memory, Destination Memory, Subscription Memory > >>> >> > >>> >> ... > >>> >> > >>> >> A destination, when it’s created, will create its own SystemUsage > >>> object > >>> >> (which creates its own separate Memory, Store, and Temp Usage > >>> objects) > >>> >> but > >>> >> it will set its parent to the be broker’s main SystemUsage object. A > >>> >> destination can have its memory limits tuned individually (but not > >>> Store > >>> >> and > >>> >> Temp, those will still delegate to the parent). To set a > >>> destination’s > >>> >> memory limit: > >>> >> ... > >>> >> > >>> >> So the destination usage objects can be used to more finely control > >>> >> MemoryUsage, but it will always coordinate with the Main memory for > >>> all > >>> >> usage counts. This functionality can be used to limit the number of > >>> >> messages > >>> >> that a destination keeps around so that a single destination cannot > >>> >> starve > >>> >> other destinations. *For queues, it also affects the store cursor’s > >>> high > >>> >> water mark. A queue has different cursors for persistent and > >>> >> non-persistent > >>> >> messages. If we hit the high water mark (a threshold of the > >>> destination’s > >>> >> memory limit), no more messages be cached ready to be dispatched, > and > >>> >> non-persistent messages can be purged to temp disk as necessary (if > >>> the > >>> >> StoreCursor will use FilePendingMessageCursor… otherwise it will > just > >>> use > >>> >> a > >>> >> VMPendingMessageCursor and won’t purge to temporary store).* > >>> >> > >>> >> If you don’t specify a memory limit for individual destinations, the > >>> >> destination’s SystemUsage will delegate to the parent (Main > >>> SystemUsage) > >>> >> for > >>> >> all usage counts. This means it will effectively use the broker’s > >>> Main > >>> >> SystemUsage for all memory-related counts. > >>> >> > >>> >> *Consumer subscriptions, on the other hand, don’t have any notion of > >>> >> their > >>> >> own SystemUsage or MemoryUsage counters. They will always use the > >>> >> broker’s > >>> >> Main SystemUsage objects.* The main thing to note about this is when > >>> >> using > >>> >> a > >>> >> FilePendingMessageCursor for subscriptions (for example, for a Topic > >>> >> subscription), the messages will not be swapped to disk until the > >>> cursor > >>> >> high-water mark (70% by default) is reached.. but that means 70% of > >>> Main > >>> >> memory will need to be reached. That could be a while, and a lot of > >>> >> messages > >>> >> could be kept in memory. And if your subscription is the one holding > >>> most > >>> >> of > >>> >> those messages, swapping to disk could take a while. As topics > >>> dispatch > >>> >> messages to one subscription at a time, if one subscription grinds > to > >>> a > >>> >> halt > >>> >> because it’s swapping its messages to disk, the rest of the > >>> subscription > >>> >> ready to receive the message will also feel the slow down..." > >>> >> > >>> >> What I don't understand from the above is why consumer subscriptions > >>> (and > >>> >> their cursors) of topics are not using per-destination MemoryLimit > >>> but > >>> >> share > >>> >> the Broker's main MemoryLimit, unlike the queues (please see the > >>> bolded > >>> >> sentences above). Due to this + some logics inside > >>> AbstractStoreCursor's > >>> >> space checking logics, PFC always kicks in for the topic whenever > the > >>> >> persistent store cursor's cache (pendingList) gets full. Let me > >>> explain > >>> >> what's happening: > >>> >> > >>> >> 1. A topic is created, with per-destination memory limit = 1MB and > >>> >> broker's > >>> >> main memory limit = 5MB. > >>> >> 2. Topic's SystemUsage (for PFC purpose) is configured with > >>> >> per-destination > >>> >> memory limit (1MB), while TopicStorePrefetch(persistent store > >>> cursor)'s > >>> >> SystemUsage (for remaining cache space checking purpose) is > >>> configured > >>> >> with > >>> >> broker's main memory limit (10MB). > >>> >> 3. The 1st message of 0.8MB is published: > >>> >> a. PFC doesn't kick in at Topic layer, as > Topic#memoryUsage#isFull > >>> is > >>> >> true. > >>> >> b. It's cached to the persistent cursor, as > >>> >> AbstractStoreCursor#hasSpace > >>> >> is true. > >>> >> c. per-dest memory usage % becomes 90%, while main memory usage % > >>> >> becomes > >>> >> 9%. > >>> >> 4. The 2nd message of 0.8MB is published: > >>> >> a. PFC doesn't kick in either, as per-dest memory usage is 90% (< > >>> >> 100%). > >>> >> This is fine. > >>> >> b. *(I expect here that this second message shouldn't be cached > to > >>> the > >>> >> cursor but invalidate it, but)this second message is also cached to > >>> the > >>> >> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% > >>> < > >>> >> 70% > >>> >> (main memory usage high watermark)!!!* > >>> >> c. per-dest memory usage % becomes 180%, while main memory usage > % > >>> >> becomes 18%. > >>> >> 5. The 3rd message of 0.8MB is published: > >>> >> a. *(I don't want this behavior but) PFC kicks in!!!* because > >>> >> per-destination memory limit is exceeded (180%). > >>> >> > >>> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks > >>> >> cursor > >>> >> memory availability against per-dest memory limit (not against > >>> broker's > >>> >> main > >>> >> memory limit) so that cursor pending cache gets disabled (just keeps > >>> what > >>> >> has been cached) w/o increasing the per-destination memory usage. > >>> >> > >>> >> I don't want to block persistent message publishing to topics by the > >>> >> per-destination memory limit but only for non-persistent message > >>> >> publishing, > >>> >> cause the store size allowed for a destination is relatively higher > >>> than > >>> >> the > >>> >> memory limit: e.g. I'd like to allow per-destination persistent > >>> message > >>> >> publishing up to 10GB disk space without being blocked by the memory > >>> >> limit > >>> >> & > >>> >> PFC... How can I achieve this? > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> View this message in context: > >>> >> > >>> > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html > >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >>> >> > >>> > > >>> > > >>> > > >>> > -- > >>> > *Christian Posta* > >>> > http://www.christianposta.com/blog > >>> > twitter: @christianposta > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> View this message in context: > >>> > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html > >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >>> > >> > >> > >> > >> -- > >> *Christian Posta* > >> http://www.christianposta.com/blog > >> twitter: @christianposta > > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4670001.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
