If there are transactions (and persistent messages), you'd actually get redelivered messages, not lost messages, when you restart the broker. Neither one is ideal, but it might be less of a problem than lost messages.
Are there other relevant log lines on either broker prior to the one you quoted? For example, are there any messages about closing the connection due to inactivity, etc.? Also, does netstat show the connection as established from both hosts, or only from one? The TCP protocol doesn't always make it easy to detect whether a connection has been closed by the other side, so seeing the connection is established from one side doesn't necessarily mean it's considered established on the other. Tim On Thu, Jul 9, 2015 at 1:30 AM, Zoltán Nyári <nyari.zol...@smartfront.hu> wrote: > Hi, > > We have a hub-and-spoke style multibroker network running activemq-5.9.1 on > ubuntu linux and openjdk 7. They are connected on WAN. > > If there is a network error, the brokers reconnect automatically, but > sometimes the DemandForwardingBridge stops working, the messages don't get > published from one-node to the other. It can be seen from the output of > 'netstat' that the tcp connection is established. > > The restarting of the broker solves the problem, but it is not always a > good solution, because we would lose messages if there are transactions > during the restart. > > I think this is the relevant exception from the activemq.log: > > WARN | Caught an exception processing local command | > org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ > BrokerService[BALA > java.net.SocketException: Socket closed > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:121) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at > > org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) > at java.io.DataOutputStream.flush(DataOutputStream.java:123) > at > > org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176) > at > > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:304) > at > > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) > at > > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > at > > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > > org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81) > at > > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:968) > at > > org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:177) > at > > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > at > > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > at > > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:138) > at > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:127) > at > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:104) > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > at > > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) > at > > org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1365) > at > > org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:884) > at > > org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:930) > at > > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Do you have any idea what is the problem? > > Thanks, > > Zoltan Nyari >
