I changed my consumers to use client-acknowledge and immediately acknowledge
the message upon receipt. My handler does the following:
void Handle(IMessage message)
{
log message
acknowledge message
Thread.Sleep(random amount of time);
}
I bound the handler to consumer.Listen. I started two processes last night,
one that sleeps randomly from 0 to 1,000ms, and another that sleeps randomly
from 0 to 30,000ms.
This morning, when I got in, the process that sleeps for up to 1000ms is
idle, and the process that sleeps for up to 30,000ms is still handling
messages.
I attached the debugger to the idle process and it has three active
threads. One is in my code where I sit in a while(true) { Thread.Sleep(); }
loop to prevent the application from exiting, and the other two are ActiveMQ
threads.
Thead appears to be blocked trying to read binary data from the tcp stream :
* OpenWireBinaryReader.cs line 132
* OpenWireFormat.cs line 165
* TcpTransport.cs line 266
Thread two is blocked waiting for an AutoResetEvent:
* DispatchingThread.cs line 125
I opened up jconsole and took a look at the ActiveMQ process but I'm no
expert with that and couldn't find anything obviously wrong. I'm at a loss
what to do next.
Thanks,
Bryan
On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <[EMAIL PROTECTED]> wrote:
> I read more about calling acknowledge, and as far as I can tell, because I
> was using transactional acknowledgement as well as session commit and
> rollback, the call to acknowledge would have been a no-op.
> Now, some of my transactions are in fact very long transactions. We have
> two sets of services, one that sits in our data center near our database,
> and another one that sits out on Amazon's EC2 cloud processing audio and
> video files and syncing them up to S3. The service by our database handles
> messages very quickly and never takes longer than a second (in theory),
> however, it often sends out new messages as part of the transaction (which
> I'll get to below).
>
> The services out on EC2 can take minutes to execute and hold the
> transaction open the whole time. It may very well be the case that ActiveMQ
> thinks my consumers are slow and is throttling them.
>
> The database handlers fail more often than the EC2 handlers. I found this
> odd at first, until I added some performance metrics to our tracing. It
> turns out (on my local machine at least) that establishing a connection can
> take upwards of 4.5 seconds. We establish a new connection and a new
> session every single time we send a message out. Most of the handlers on
> our database side send out at least one message, so the transactions are
> held open for at least that long.
>
> So, it's obvious to me that there's no connection pooling going on and
> reading the FAQ confirms this. Assuming there was connection pooling was
> clearly a bad assumption on my part. We don't use spring internally, so we
> have no pre-built components available to handle connection pooling. For
> now, I'm going to deal with it by simply moving away from transactional
> message processing, and immediately acknowledging every message when it is
> received. I'll deal with the consequences of failing message handlers
> later.
>
> I was also consuming messages in a very different fashion last time I used
> a non-transactional method, so perhaps this change in combination with that
> previous change will get my system into working order. Once again, I'll
> post back here with the results of these changes.
>
> If you notice any flaw in my reasoning, please do tell me. :)
>
> Thanks,
> Bryan
>
>
