Maybe some more logging will help identify the root cause. Enable TRACE level logging for the following packages: org.apache.activemq.broker.region org.apache.activemq.network org.apache.activemq.command
and open a ticket and attach the log if you can still reproduce. On 17 May 2012 23:32, Chris Robison <chrisdrobi...@gmail.com> wrote: > I've been able to successfully reproduce the issue in a dev environment, > but it's hard to identity exactly what is causing it. It is almost like it > is a race condition of some kind. > > On Wed, May 16, 2012 at 10:27 PM, Gary Tully <gary.tu...@gmail.com> wrote: > >> sorry, yes, there is one more piece to the puzzle. The temp >> destinations need to have a well known naming convention and there >> needs to be a wildcard static include in the network bridge to have >> matching destinations propagated. >> >> Have a look at the test case to see an example of the xml config required: >> >> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java >> >> In that code, note the use of: >> factory.setConnectionIDPrefix(...) to specify a prefix for all temp >> destinations that can be used for the wildcard static include >> <staticallyIncludedDestinations> >> >> This is needed because the destination advisories are no longer propagated. >> >> On 16 May 2012 18:13, Chris Robison <chrisdrobi...@gmail.com> wrote: >> > So I tried that and the client appears to be sending just fine, but the >> > producer on the other side of the network connector that expects to hear >> > back via the temp queue is not receiving the message. >> > >> > Chris >> > >> > On Wed, May 16, 2012 at 1:27 AM, Gary Tully <gary.tu...@gmail.com> >> wrote: >> > >> >> you need one additional bit of config, to disable the connection from >> >> tracking temp destination advisories so that it allows the request to >> >> go to the broker. >> >> >> >> factory.setWatchTopicAdvisories(false); >> >> or ?jms.watchTopicAdvisories=false on the connection factory broker url >> >> >> >> On 15 May 2012 23:30, Chris Robison <chrisdrobi...@gmail.com> wrote: >> >> > I tried that last suggestion and it appears as though >> >> > the allowTempAutoCreationOnSend is not working. The client keep >> >> receiving a >> >> > "cannot send to a deleted destination" error. >> >> > >> >> > Chris >> >> > >> >> > On Mon, May 14, 2012 at 4:19 PM, Gary Tully <gary.tu...@gmail.com> >> >> wrote: >> >> > >> >> >> this is expected, but the flooding impact should be minimal with a >> >> >> small number of brokers in the network and a reasonable number of >> >> >> connections. >> >> >> >> >> >> each network bridge will register interest in all temp destination >> >> >> advisories, so that it will be in a position to receive messages for >> >> >> those destinations. It can't know in advance which messages may be >> >> >> routed though it, so it needs to react to each advisory message and >> >> >> create a local temp destination. When the connection that creates the >> >> >> temp is closes, the temp destination gets closed and each broker >> >> >> reacts to that event via the remove temp advisory. >> >> >> >> >> >> Temp destinations are often pooled for serial request reply messages >> >> >> or multiplexed using selectors, to reduce the overhead of creation >> and >> >> >> propagation, in a network scenario. The other option is to use >> regular >> >> >> destinations for replys >> >> >> >> >> >> For temp destinations, there is an alternative solution that would >> >> >> require a little more configuration but which would reduce the >> >> >> overhead of unnecessary temp destination propagation. >> >> >> You can disable auto temp destination propagation through the network >> >> >> bridge, ie: bridgeTempDestinations=false. >> >> >> Then you need to allow temp destination creation on the fly for a >> >> >> producer, broker.allowTempAutoCreationOnSend=true >> >> >> To finish, you need to configure >> >> >> broker.schedulePeriodForDestinationPurge>0 to have the on the fly >> >> >> created temp destinations purged when they are empty and enable it >> for >> >> >> temp destinations. policy entry gcInactiveDestinations=true (this is >> >> >> important because the auto created temp destination cannot be >> >> >> associated with a connection in any sensible way) >> >> >> >> >> >> There is a test that exercises these options >> >> >> >> >> >> >> >> >> http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java >> >> >> >> >> >> >> >> >> On 11 May 2012 17:45, Chris Robison <chrisdrobi...@gmail.com> wrote: >> >> >> > Another interesting observation: as a test, I connected all >> producers >> >> and >> >> >> > consumers to MSTMIP102 (the machine that doesn't have the network >> >> >> > connector). I then started MSIPAP101 so that its network connector >> >> could >> >> >> > start and connect to MSTMIP102. Request request-reply works fine >> >> between >> >> >> > all producers and consumers, but after 1 or 2 request-replies, >> >> MSIPAP101 >> >> >> is >> >> >> > flooded with the temp queue advisories--even though there are no >> >> >> producers >> >> >> > or consumers connected to it. >> >> >> > >> >> >> > On Fri, May 11, 2012 at 10:09 AM, Chris Robison < >> >> chrisdrobi...@gmail.com >> >> >> >wrote: >> >> >> > >> >> >> >> Here is the log from the latest run with the TTL down to 1. It >> still >> >> >> >> floods after about 1 or 2 request-reply calls. It just seems to be >> >> >> >> alternating between add temp queue and remove temp queue messages. >> >> >> >> >> >> >> >> On Fri, May 11, 2012 at 9:52 AM, Chris Robison < >> >> chrisdrobi...@gmail.com >> >> >> >wrote: >> >> >> >> >> >> >> >>> Yes, because we have a spoke and hub topology where a client in >> one >> >> >> spoke >> >> >> >>> may need to request info from a client connected to a different >> >> spoke. >> >> >> I >> >> >> >>> suppose I could lower that value. >> >> >> >>> >> >> >> >>> >> >> >> >>> On Fri, May 11, 2012 at 9:47 AM, Gary Tully < >> gary.tu...@gmail.com> >> >> >> wrote: >> >> >> >>> >> >> >> >>>> Do you have a good reason to have networkTTL="5", if there is >> only >> >> one >> >> >> >>>> broker, the default value of 1 will be fine. >> >> >> >>>> >> >> >> >>>> On 11 May 2012 16:32, Chris Robison <chrisdrobi...@gmail.com> >> >> wrote: >> >> >> >>>> > If it helps, I've attached the configuration files for both >> >> >> machines. >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > On Fri, May 11, 2012 at 9:07 AM, Chris Robison < >> >> >> >>>> chrisdrobi...@gmail.com> >> >> >> >>>> > wrote: >> >> >> >>>> >> >> >> >> >>>> >> One thing I have noticed though is that when I restart the >> >> broker >> >> >> on >> >> >> >>>> the >> >> >> >>>> >> other end, everything starts to work again until it floods >> >> again. >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> On Fri, May 11, 2012 at 9:05 AM, Chris Robison < >> >> >> >>>> chrisdrobi...@gmail.com> >> >> >> >>>> >> wrote: >> >> >> >>>> >>> >> >> >> >>>> >>> I'm on 5.6. And all brokers remain up and running. >> >> >> >>>> >>> >> >> >> >>>> >>> >> >> >> >>>> >>> On Fri, May 11, 2012 at 8:48 AM, Gary Tully < >> >> gary.tu...@gmail.com >> >> >> > >> >> >> >>>> wrote: >> >> >> >>>> >>>> >> >> >> >>>> >>>> what version are you on? >> >> >> >>>> >>>> Is there any chance that the broker at the other end of the >> >> >> network >> >> >> >>>> >>>> bridge is shutting down? >> >> >> >>>> >>>> >> >> >> >>>> >>>> On 11 May 2012 14:34, Chris Robison < >> chrisdrobi...@gmail.com> >> >> >> >>>> wrote: >> >> >> >>>> >>>> > I have a network of brokers and I am using a >> request-reply >> >> >> system >> >> >> >>>> >>>> > across >> >> >> >>>> >>>> > that network, but I've noticed that after a few >> >> request-replys, >> >> >> >>>> the >> >> >> >>>> >>>> > broker >> >> >> >>>> >>>> > network will get flooded with TempQueue advisory >> messages, >> >> then >> >> >> >>>> some >> >> >> >>>> >>>> > of the >> >> >> >>>> >>>> > of the brokers will be reported as slow consumers and the >> >> >> >>>> consumers >> >> >> >>>> >>>> > won't >> >> >> >>>> >>>> > receive messages after that. The advisory messages seem >> to >> >> be >> >> >> >>>> repeated >> >> >> >>>> >>>> > (e.g. add temp destination, remove temp destination) for >> the >> >> >> same >> >> >> >>>> temp >> >> >> >>>> >>>> > destinations. Not quite sure how to handle this. Any >> >> >> suggestions? >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > Here are some of the errors are other things I'm seeing >> >> >> >>>> repeatedly in >> >> >> >>>> >>>> > logs >> >> >> >>>> >>>> > when I enable debug. In fact, when this flood occurs, >> tons >> >> of >> >> >> log >> >> >> >>>> gets >> >> >> >>>> >>>> > generated all at once. >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,248 | DEBUG | Error occured while >> >> >> processing >> >> >> >>>> async >> >> >> >>>> >>>> > command: DestinationInfo {commandId = 40578, >> >> responseRequired = >> >> >> >>>> false, >> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, >> >> >> destination = >> >> >> >>>> >>>> > >> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, >> >> >> >>>> >>>> > operationType = >> >> >> >>>> >>>> > 0, timeout = 0, brokerPath = >> >> >> >>>> [ID:MSSD5D101-38580-1336708634042-0:1]}, >> >> >> >>>> >>>> > exception: java.lang.IllegalStateException: Timer already >> >> >> >>>> cancelled. | >> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | >> >> >> ActiveMQ >> >> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493 >> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. >> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397) >> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) >> >> >> >>>> >>>> > at >> >> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,248 | WARN | Async error occurred: >> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already >> cancelled. | >> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | >> >> >> ActiveMQ >> >> >> >>>> >>>> > Transport: ssl:///10.60.252.28:54493 >> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. >> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397) >> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) >> >> >> >>>> >>>> > at >> >> >> org.apache.activemq.broker.region.Queue.start(Queue.java:889) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,249 | DEBUG | mstmip102.vosh.netadding >> >> >> >>>> >>>> > destination: >> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.Connection | >> >> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion | >> >> >> StopAsync:vm:// >> >> >> >>>> >>>> > mstmip102.vosh.net#0-23 >> >> >> >>>> >>>> > 2012-05-11 05:24:37,250 | WARN | Failed to remove >> >> connection >> >> >> >>>> >>>> > ConnectionInfo {commandId = 3, responseRequired = false, >> >> >> >>>> connectionId >> >> >> >>>> >>>> > = >> >> >> >>>> >>>> > ID:msipap102-50908-1336690011745-7:1, clientId = >> >> >> >>>> >>>> > mstmip102_msipap102.vosh.net_outbound, clientIp = >> >> >> >>>> >>>> > vm://mstmip102.vosh.net#0, >> >> >> >>>> >>>> > userName = mqipauser, password = *****, brokerPath = >> null, >> >> >> >>>> >>>> > brokerMasterConnector = false, manageable = false, >> >> >> clientMaster = >> >> >> >>>> >>>> > true, >> >> >> >>>> >>>> > faultTolerant = false, failoverReconnect = false}, >> reason: >> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already >> cancelled. | >> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | >> >> >> >>>> >>>> > StopAsync:vm:// >> >> >> >>>> >>>> > mstmip102.vosh.net#0-23 >> >> >> >>>> >>>> > 2012-05-11 05:24:37,250 | DEBUG | Exception detail: | >> >> >> >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | >> >> >> >>>> >>>> > StopAsync:vm:// >> >> >> >>>> >>>> > mstmip102.vosh.net#0-23 >> >> >> >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. >> >> >> >>>> >>>> > at java.util.Timer.sched(Timer.java:397) >> >> >> >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) >> >> >> >>>> >>>> > at >> >> >> org.apache.activemq.broker.region.Topic.start(Topic.java:530) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:286) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:481) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:533) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:464) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:459) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:234) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:351) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.security.JaasAuthenticationBroker.removeConnection(JaasAuthenticationBroker.java:95) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:766) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068) >> >> >> >>>> >>>> > at >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > >> >> >> >>>> >> >> >> >> >> >> org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996) >> >> >> >>>> >>>> > at java.lang.Thread.run(Thread.java:722) >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,252 | DEBUG | mstmip102.vosh.netadding >> >> >> >>>> >>>> > destination: >> >> >> >>>> >>>> > >> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1 | >> >> >> >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion | >> ActiveMQ >> >> >> >>>> Transport: >> >> >> >>>> >>>> > ssl:///172.17.5.78:59658 >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging >> >> (mstmip102.vosh.net-> >> >> >> >>>> >>>> > msipap101.vosh.net) >> >> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184, >> >> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, >> >> destination >> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath: >> >> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: >> >> >> ActiveMQMessage >> >> >> >>>> >>>> > {commandId >> >> >> >>>> >>>> > = 0, responseRequired = false, messageId = >> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184, >> >> >> >>>> originalDestination = >> >> >> >>>> >>>> > null, >> >> >> >>>> >>>> > originalTransactionId = null, producerId = >> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination = >> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = >> null, >> >> >> >>>> expiration >> >> >> >>>> >>>> > = 0, >> >> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588, >> >> >> >>>> >>>> > brokerOutTime = >> >> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null, >> >> >> persistent = >> >> >> >>>> >>>> > false, >> >> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null, >> >> groupSequence = >> >> >> 0, >> >> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID = >> null, >> >> >> >>>> content = >> >> >> >>>> >>>> > null, >> >> >> >>>> >>>> > marshalledProperties = >> >> >> >>>> org.apache.activemq.util.ByteSequence@567edec8, >> >> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0, >> >> >> responseRequired = >> >> >> >>>> >>>> > false, >> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, >> >> >> destination = >> >> >> >>>> >>>> > >> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, >> >> >> >>>> >>>> > operationType = >> >> >> >>>> >>>> > 0, timeout = 0, brokerPath = null}, redeliveryCounter = >> 0, >> >> >> size = >> >> >> >>>> >>>> > 1024, >> >> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616, >> >> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, >> >> >> >>>> originBrokerName= >> >> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false, >> >> readOnlyBody >> >> >> = >> >> >> >>>> false, >> >> >> >>>> >>>> > droppable = false} | >> >> >> >>>> >>>> > >> org.apache.activemq.network.DemandForwardingBridgeSupport | >> >> >> >>>> ActiveMQ >> >> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4 >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging >> >> (mstmip102.vosh.net-> >> >> >> >>>> >>>> > msipap101.vosh.net) >> >> >> >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185, >> >> >> >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, >> >> destination >> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath: >> >> >> >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: >> >> >> ActiveMQMessage >> >> >> >>>> >>>> > {commandId >> >> >> >>>> >>>> > = 0, responseRequired = false, messageId = >> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185, >> >> >> >>>> originalDestination = >> >> >> >>>> >>>> > null, >> >> >> >>>> >>>> > originalTransactionId = null, producerId = >> >> >> >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination = >> >> >> >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = >> null, >> >> >> >>>> expiration >> >> >> >>>> >>>> > = 0, >> >> >> >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588, >> >> >> >>>> >>>> > brokerOutTime = >> >> >> >>>> >>>> > 1336713877341, correlationId = null, replyTo = null, >> >> >> persistent = >> >> >> >>>> >>>> > false, >> >> >> >>>> >>>> > type = Advisory, priority = 0, groupID = null, >> >> groupSequence = >> >> >> 0, >> >> >> >>>> >>>> > targetConsumerId = null, compressed = false, userID = >> null, >> >> >> >>>> content = >> >> >> >>>> >>>> > null, >> >> >> >>>> >>>> > marshalledProperties = >> >> >> >>>> org.apache.activemq.util.ByteSequence@235448d5, >> >> >> >>>> >>>> > dataStructure = DestinationInfo {commandId = 0, >> >> >> responseRequired = >> >> >> >>>> >>>> > false, >> >> >> >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, >> >> >> destination = >> >> >> >>>> >>>> > >> temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, >> >> >> >>>> >>>> > operationType = >> >> >> >>>> >>>> > 1, timeout = 0, brokerPath = null}, redeliveryCounter = >> 0, >> >> >> size = >> >> >> >>>> >>>> > 1024, >> >> >> >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616, >> >> >> >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, >> >> >> >>>> originBrokerName= >> >> >> >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false, >> >> readOnlyBody >> >> >> = >> >> >> >>>> false, >> >> >> >>>> >>>> > droppable = false} | >> >> >> >>>> >>>> > >> org.apache.activemq.network.DemandForwardingBridgeSupport | >> >> >> >>>> ActiveMQ >> >> >> >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4 >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 13:29:40,053 | WARN | TopicSubscription: >> >> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-4:2:1:1, >> >> >> destinations=7, >> >> >> >>>> >>>> > dispatched=1000, delivered=7723, matched=1001, >> discarded=0: >> >> has >> >> >> >>>> twice >> >> >> >>>> >>>> > its >> >> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be >> >> slow | >> >> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription | >> >> ActiveMQ >> >> >> >>>> >>>> > Transport: >> >> >> >>>> >>>> > ssl:///10.60.252.28:54614 >> >> >> >>>> >>>> > >> >> >> >>>> >>>> > 2012-05-11 13:29:42,540 | WARN | TopicSubscription: >> >> >> >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-3:2:1:1, >> >> >> destinations=6, >> >> >> >>>> >>>> > dispatched=1000, delivered=9922, matched=1001, >> discarded=0: >> >> has >> >> >> >>>> twice >> >> >> >>>> >>>> > its >> >> >> >>>> >>>> > prefetch limit pending, without an ack; it appears to be >> >> slow | >> >> >> >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription | >> >> ActiveMQ >> >> >> >>>> >>>> > Transport: >> >> >> >>>> >>>> > ssl:///10.60.252.28:54614 >> >> >> >>>> >>>> >> >> >> >>>> >>>> >> >> >> >>>> >>>> >> >> >> >>>> >>>> -- >> >> >> >>>> >>>> http://fusesource.com >> >> >> >>>> >>>> http://blog.garytully.com >> >> >> >>>> >>> >> >> >> >>>> >>> >> >> >> >>>> >> >> >> >> >>>> > >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> -- >> >> >> >>>> http://fusesource.com >> >> >> >>>> http://blog.garytully.com >> >> >> >>>> >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> http://fusesource.com >> >> >> http://blog.garytully.com >> >> >> >> >> >> >> >> >> >> >> -- >> >> http://fusesource.com >> >> http://blog.garytully.com >> >> >> >> >> >> -- >> http://fusesource.com >> http://blog.garytully.com >> -- http://fusesource.com http://blog.garytully.com
