As I understand it, the cursor spools out to disk messages that have been committed into the memory store. If they're participating in a transaction, they're not committed and are instead just sitting in memory in the org.apache.activemq.store.memory.MemoryTransactionStore. (I assume we're talking about non-persistent messages here, since persistent ones are written to a disk-based store to begin with, so spooling via a cursor isn't a factor there.) If you look at the code for org.apache.activemq.store.memory.MemoryTransactionStore.Tx, you'll see that the messages are stored in a List<AddMessageCommand>, which means they're simply stored on the heap and not eligible to be spooled automatically.
Here are a few things you could do: 1. You could modify org.apache.activemq.store.memory.MemoryTransactionStore.Tx to do your own spooling, perhaps to a KahaDBStore objecct you would create for that purpose. 2. You could increase the size of the broker's heap. 1GB is nothing by today's standards, especially if you're expecting to be using more than that from time to time. Throwing more heap space at the problem could be a perfectly viable option if you're able to place an upper limit on how much heap might be used by a single transaction and then size the broker much larger than that. 3. If you're not able to get enough RAM to handle the heap size you'd need, you might try allocating (at the operating system) some swap space to be used as virtual memory, and then setting the -Xmx argument to a large enough value. I can't say for sure that this will work (I'm not sure the JVM will be willing to allocate a heap that spans both RAM and swap), but you could easily test by setting -Xms to the same value as -Xmx. But be aware that the OS may be considerably less performant when using swap (because disk is much slower than RAM) and that Linux in particular may not choose to stop using swap even if it gets back to a state where it could fit everything in RAM, so there are significant potential disadvantages to this approach even if it works. 4. Get your producers to send fewer messages in a transaction. 5. Get your producers to send smaller messages in a transaction (e.g. by storing the content externally, such as in a cache, and then just passing the key rather than the full content). Hope this help. Tim On Fri, Mar 9, 2018 at 12:23 PM, bmadaras <bryan.mada...@armssoftware.com> wrote: > I have been able to successfully have large number of persistent messages > spool over to the filesystem exactly as intended by using the > cursorMemoryHighWaterMark for messages that have been enqueued to the > queue. > > The issue I am running into, it happens only a few times per day in our > system, is that a large transaction is created that tries to enqueue a > couple hundred thousand messages and during the transaction none of the > rules in place seem to cause the messages to get spooled over to disk. If > the number of messages is large enough the broker can still run out of > memory. > > For this scenario I am not too worried about speed as much as I am about > keeping the broker alive to receive and store the messages. > > I am using the file based cursor as I had noticed during some larger > transactions that was causing the queue to hit the > cursorMemoryHighWaterMark > messages that had been committed to the queue were getting blocked behind > the larger transaction which does not seem to occur with the file based > cursor. > > Any thoughts on how to handle this scenario would be greatly appreciated. > > The broker is started with a gig of heap space. -Xmx1G > <destinationPolicy> > <policyMap> > <policyEntries> > > <policyEntry topic=">" > producerFlowControl="true"> > > <pendingMessageLimitStrategy> > > <constantPendingMessageLimitStrategy > > limit="1000" > /> > > </pendingMessageLimitStrategy> > </policyEntry> > <policyEntry queue=">" > producerFlowControl="false" cursorMemoryHighWaterMark="10"> > > > > <pendingQueuePolicy><fileQueueCursor/></pendingQueuePolicy> > > </policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="64 mb" /> > </memoryUsage> > <storeUsage> > <storeUsage limit="30 gb" /> > </storeUsage> > <tempUsage> > <tempUsage limit="2 gb" /> > </tempUsage> > </systemUsage> > </systemUsage> > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
