More information...
There were ~ 4,000 messages still pending in the queue. I took a closer
look at the two active consumers under jconsole, the consumer that was
running had approximately 800 messages in it's
MessageCountAwaitingAcknowledge, the consumer that was idle had 0.
I put a few new messages into the message queue, and this caused the idle
process to unblock and start processing messages. Whatever this did, it
unstuck the idle process and I now have significantly less than 4,000
messages and it's continuing to tick downwards.
It's pretty clear that ActiveMQ decided to stop sending messages to my
consumer.
Bryan
On Fri, Aug 29, 2008 at 9:40 AM, Bryan Murphy <[EMAIL PROTECTED]> wrote:
> I changed my consumers to use client-acknowledge and immediately
> acknowledge the message upon receipt. My handler does the following:
>
> void Handle(IMessage message)
> {
> log message
> acknowledge message
> Thread.Sleep(random amount of time);
> }
>
> I bound the handler to consumer.Listen. I started two processes last
> night, one that sleeps randomly from 0 to 1,000ms, and another that sleeps
> randomly from 0 to 30,000ms.
>
> This morning, when I got in, the process that sleeps for up to 1000ms is
> idle, and the process that sleeps for up to 30,000ms is still handling
> messages.
>
> I attached the debugger to the idle process and it has three active
> threads. One is in my code where I sit in a while(true) { Thread.Sleep(); }
> loop to prevent the application from exiting, and the other two are ActiveMQ
> threads.
>
> Thead appears to be blocked trying to read binary data from the tcp stream
> :
> * OpenWireBinaryReader.cs line 132
> * OpenWireFormat.cs line 165
> * TcpTransport.cs line 266
>
> Thread two is blocked waiting for an AutoResetEvent:
> * DispatchingThread.cs line 125
>
> I opened up jconsole and took a look at the ActiveMQ process but I'm no
> expert with that and couldn't find anything obviously wrong. I'm at a loss
> what to do next.
>
> Thanks,
> Bryan
>
> On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <[EMAIL PROTECTED]>wrote:
>
>> I read more about calling acknowledge, and as far as I can tell, because I
>> was using transactional acknowledgement as well as session commit and
>> rollback, the call to acknowledge would have been a no-op.
>> Now, some of my transactions are in fact very long transactions. We have
>> two sets of services, one that sits in our data center near our database,
>> and another one that sits out on Amazon's EC2 cloud processing audio and
>> video files and syncing them up to S3. The service by our database handles
>> messages very quickly and never takes longer than a second (in theory),
>> however, it often sends out new messages as part of the transaction (which
>> I'll get to below).
>>
>> The services out on EC2 can take minutes to execute and hold the
>> transaction open the whole time. It may very well be the case that ActiveMQ
>> thinks my consumers are slow and is throttling them.
>>
>> The database handlers fail more often than the EC2 handlers. I found this
>> odd at first, until I added some performance metrics to our tracing. It
>> turns out (on my local machine at least) that establishing a connection can
>> take upwards of 4.5 seconds. We establish a new connection and a new
>> session every single time we send a message out. Most of the handlers on
>> our database side send out at least one message, so the transactions are
>> held open for at least that long.
>>
>> So, it's obvious to me that there's no connection pooling going on and
>> reading the FAQ confirms this. Assuming there was connection pooling was
>> clearly a bad assumption on my part. We don't use spring internally, so we
>> have no pre-built components available to handle connection pooling. For
>> now, I'm going to deal with it by simply moving away from transactional
>> message processing, and immediately acknowledging every message when it is
>> received. I'll deal with the consequences of failing message handlers
>> later.
>>
>> I was also consuming messages in a very different fashion last time I used
>> a non-transactional method, so perhaps this change in combination with that
>> previous change will get my system into working order. Once again, I'll
>> post back here with the results of these changes.
>>
>> If you notice any flaw in my reasoning, please do tell me. :)
>>
>> Thanks,
>> Bryan
>>
>>
