the fix can be seen in the linked subversion commits tab of the issue: https://issues.apache.org/activemq/browse/AMQ-2100?page=com.atlassian.jira.plugin.ext.subversion:subversion-commits-tabpanel click on the modified files links to see the diffs.
2009/2/10 bonnyr <bon...@optusnet.com.au>: > > Gary, > > Agreed about the test case - my familiarity with the AMQ test suite is not > that good (excuses, excuses :) ) so what I tried to do, unsuccessfully, was > to play with the ConsumerTool without much luck > on my dev machine which is a WinXP running on a Centrino Duo, and which is > different from our target > boxes - SunFire 420s. Scheduling is possibly different? > > In any case, I'm glad you've been able to reproduce this. Is the fix going > to be linked to the JIRA item? > If so, I would be able to apply it (in case it really is localised) to our > version of AMQ and continue > with out tests. > > Cheers, > > Bonny > > > Gary Tully wrote: >> >> Bonnie, >> in CLIENT_ACK, the list is used on close to deal with duplicate >> suppression, these messages will >> get redelivered so they need to be removed from audit that tracks >> their receipt. That logic is at fault. >> >> Cheers, >> Gary. >> >> 2009/2/9 bonnyr <bon...@optusnet.com.au>: >>> >>> Gary, >>> >>> Why would the collection be used in CLIENT_ACK mode then? >>> >>> In our application we've got to work in this mode since there's a >>> processing >>> chain >>> that may take a while between delivery of the message and its ack, so >>> chaging this >>> mode is not possible for our use case. >>> >>> We're aware of the fact that CLIENT_ACK actually acks all the messages in >>> a >>> session and >>> it's not really related to the message itself, but that's a story for >>> another day and another >>> version of AMQ... >>> >>> However, wouldn't static analysis of the code be sufficient in this case? >>> The code segment >>> above is the only thing that is not protected by synchronization, whilst >>> every other use of >>> this collection is. >>> >>> Cheers, >>> >>> Bonny >>> >>> >>> Gary Tully wrote: >>>> >>>>> I'm not quite able to cause this to happen in a simple test case. >>>>> Perhaps >>>>> this is to do with some >>>>> thread timing/scheduling issues in the simple test case vs our >>>>> application. >>>>> >>>> Yea, that is fairly typical. >>>> >>>>> However, are you in a >>>>> position to explain what the deliveredMessage collection is used for? >>>>> Is >>>>> it >>>>> not used in all connection/ >>>>> ack modes? >>>>> >>>> Sure, the deliveredMessage list is used to generate acks in the Auto >>>> ack mode. In addition, >>>> with the consumer optimizeAcknowledge option, there is more likely to >>>> be some outstanding >>>> delivered messages that require acks. On close, any outstanding acks >>>> will require delivery so >>>> you may run into contention with concurrent delivery and close >>>> execution as you suggest. >>>> >>>> May I suggest using AUTO_ACK and additionally using >>>> optimizeAcknowledge if needed. >>>> >>>>> Perhaps if I knew what the behaviour should be like, I'd be able >>>>> to construct a test >>>>> case that simulate our environment. >>>>> >>>> Hopefully :-) >>>> >>>> Gary. >>>> >>>> >>>>> Cheers, >>>>> >>>>> Bonny >>>>> >>>>> >>>>> Gary Tully wrote: >>>>>> >>>>>> Hi Bonny, >>>>>> that looks like a bug indeed, should be easy to replicate in a Junit >>>>>> tests case I think. Could you raise a jira issue for this and if you >>>>>> have some tests code that demonstrates, please include it. >>>>>> >>>>>> for more info see: http://activemq.apache.org/contributing.html >>>>>> >>>>>> Thanks, >>>>>> Gary. >>>>>> >>>>>> 2009/2/6 bonnyr <bon...@optusnet.com.au>: >>>>>>> >>>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today) >>>>>>> >>>>>>> My setup: >>>>>>> * Broker is configured with a single queue, full with messages, on a >>>>>>> host >>>>>>> accessible via the network. >>>>>>> * Application configured with a single consumer, connected to a >>>>>>> single >>>>>>> sesssion, running in its own thread. >>>>>>> * ActiveMQ delivers lots of messages using one of the >>>>>>> ActiveMQSessionTask >>>>>>> threads. >>>>>>> * Session configured as CLIENT_ACKNOLEDGE >>>>>>> >>>>>>> >>>>>>> Since the queue is full of messages, delivery of these messages >>>>>>> happen >>>>>>> as >>>>>>> fast as the network >>>>>>> can deliver and the AMQ thread is invoking the onMessage with each >>>>>>> new >>>>>>> message. These messages >>>>>>> are then processed by the consumer thread. The consumer thread then >>>>>>> decides >>>>>>> to close the connection >>>>>>> and the following ensues: >>>>>>> [code] >>>>>>> java.util.ConcurrentModificationException >>>>>>> at >>>>>>> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:617) >>>>>>> at java.util.LinkedList$ListItr.next(LinkedList.java:552) >>>>>>> at >>>>>>> org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663) >>>>>>> at >>>>>>> org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583) >>>>>>> at >>>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351) >>>>>>> at com.xxx.app.AMsgQueueConsumer.suspend >>>>>>> >>>>>>> ... snip ... >>>>>>> >>>>>>> [/code] >>>>>>> >>>>>>> This happens because AMQ is busy delivering messages and there is a >>>>>>> collection [b]deliveredMessages[/b] that is >>>>>>> not protected by synchronisation in exactly one place (everywhere >>>>>>> else >>>>>>> it >>>>>>> is...) which is executing the >>>>>>> following code (in the dispose method): >>>>>>> [code] >>>>>>> ... snip ... >>>>>>> if (session.isClientAcknowledge()) { >>>>>>> if (!this.info.isBrowser()) { >>>>>>> // rollback duplicates that aren't acknowledged >>>>>>> for (MessageDispatch old : deliveredMessages) { >>>>>>> session.connection.rollbackDuplicate(this, >>>>>>> old.getMessage()); >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> ... snip ... >>>>>>> [/code] >>>>>>> >>>>>>> Since the code iterates over the collection, and the iterator checks >>>>>>> for >>>>>>> modifications to the underlying collection, and since the AMQ message >>>>>>> delivery thread has managed to sneak in a couple more messages >>>>>>> while the consumer thread attempted to close the connection, the >>>>>>> problem >>>>>>> occurs (it could be that >>>>>>> the for syntax hides the explicit iterator calls, but...) >>>>>>> >>>>>>> >>>>>>> Is this an ommission or is there a reason for not synchronising >>>>>>> access >>>>>>> to >>>>>>> this collection? If it's not a bug >>>>>>> then how should the consumer be disconnected? >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Bonny >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html >>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> http://blog.garytully.com >>>>>> >>>>>> Open Source SOA >>>>>> http://FUSESource.com >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> http://blog.garytully.com >>>> >>>> Open Source SOA >>>> http://FUSESource.com >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> http://blog.garytully.com >> >> Open Source SOA >> http://FUSESource.com >> >> > > -- > View this message in context: > http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945059.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source SOA http://FUSESource.com
