I changed my consumers to use client-acknowledge and immediately acknowledge the message upon receipt. My handler does the following:
void Handle(IMessage message) { log message acknowledge message Thread.Sleep(random amount of time); } I bound the handler to consumer.Listen. I started two processes last night, one that sleeps randomly from 0 to 1,000ms, and another that sleeps randomly from 0 to 30,000ms. This morning, when I got in, the process that sleeps for up to 1000ms is idle, and the process that sleeps for up to 30,000ms is still handling messages. I attached the debugger to the idle process and it has three active threads. One is in my code where I sit in a while(true) { Thread.Sleep(); } loop to prevent the application from exiting, and the other two are ActiveMQ threads. Thead appears to be blocked trying to read binary data from the tcp stream : * OpenWireBinaryReader.cs line 132 * OpenWireFormat.cs line 165 * TcpTransport.cs line 266 Thread two is blocked waiting for an AutoResetEvent: * DispatchingThread.cs line 125 I opened up jconsole and took a look at the ActiveMQ process but I'm no expert with that and couldn't find anything obviously wrong. I'm at a loss what to do next. Thanks, Bryan On Thu, Aug 28, 2008 at 9:01 PM, Bryan Murphy <[EMAIL PROTECTED]> wrote: > I read more about calling acknowledge, and as far as I can tell, because I > was using transactional acknowledgement as well as session commit and > rollback, the call to acknowledge would have been a no-op. > Now, some of my transactions are in fact very long transactions. We have > two sets of services, one that sits in our data center near our database, > and another one that sits out on Amazon's EC2 cloud processing audio and > video files and syncing them up to S3. The service by our database handles > messages very quickly and never takes longer than a second (in theory), > however, it often sends out new messages as part of the transaction (which > I'll get to below). > > The services out on EC2 can take minutes to execute and hold the > transaction open the whole time. It may very well be the case that ActiveMQ > thinks my consumers are slow and is throttling them. > > The database handlers fail more often than the EC2 handlers. I found this > odd at first, until I added some performance metrics to our tracing. It > turns out (on my local machine at least) that establishing a connection can > take upwards of 4.5 seconds. We establish a new connection and a new > session every single time we send a message out. Most of the handlers on > our database side send out at least one message, so the transactions are > held open for at least that long. > > So, it's obvious to me that there's no connection pooling going on and > reading the FAQ confirms this. Assuming there was connection pooling was > clearly a bad assumption on my part. We don't use spring internally, so we > have no pre-built components available to handle connection pooling. For > now, I'm going to deal with it by simply moving away from transactional > message processing, and immediately acknowledging every message when it is > received. I'll deal with the consequences of failing message handlers > later. > > I was also consuming messages in a very different fashion last time I used > a non-transactional method, so perhaps this change in combination with that > previous change will get my system into working order. Once again, I'll > post back here with the results of these changes. > > If you notice any flaw in my reasoning, please do tell me. :) > > Thanks, > Bryan > >