I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair
of RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition
(build 1.5.0_11-b03) and it doesn't seem to reliably handle failover.
The first issue I am seeing is that the master server keeps generating the
following message in the log file:
2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
java.lang.AssertionError: Unsupported Method
at
org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
at
org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at
org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
at
org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
at org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
at org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Thread.java:595)
I'm not sure if this is a problem, but it doesn't give my any warm
fuzzies. Can this be ignored or is this a real problem?
After the slave starts up the connection between the master and the slave
times out (from the perspective of the slave), the slave tries to start up
it's connectors and fails. Here's what I'm seeing in the slave's log
file:
2007-06-18 10:04:11,735 [main ] INFO BrokerService - ActiveMQ 4.1.1
JMS Message Broker (shakira-slave) is starting
2007-06-18 10:04:11,845 [JMX connector ] INFO ManagementContext - JMX
consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
2007-06-18 10:04:12,721 [main ] INFO JDBCPersistenceAdapter -
Database driver recognized: [apache_derby_embedded_jdbc_driver]
2007-06-18 10:04:13,943 [main ] INFO DefaultDatabaseLocker -
Attempting to acquire the exclusive lock to become the Master broker
2007-06-18 10:04:13,944 [main ] INFO DefaultDatabaseLocker -
Becoming the master on dataSource: [EMAIL PROTECTED]
2007-06-18 10:04:14,012 [main ] INFO JournalPersistenceAdapter -
Journal Recovery Started from: Active Journal: using 5 x 20.0 Megs at:
/data/run/activemq-data/journal
2007-06-18 10:04:14,079 [main ] INFO JournalPersistenceAdapter -
Journal Recovered: 0 message(s) in transactions recovered.
2007-06-18 10:04:14,221 [main ] INFO TransportServerThreadSupport -
Listening for connections at: tcp://slave.cac.washington.edu:61616
2007-06-18 10:04:14,235 [main ] INFO TransportConnector - Connector
openwire Started
2007-06-18 10:04:14,244 [main ] INFO TransportServerThreadSupport -
Listening for connections at: stomp://slave.cac.washington.edu:61613
2007-06-18 10:04:14,244 [main ] INFO TransportConnector - Connector
stomp Started
2007-06-18 10:04:14,247 [main ] INFO NetworkConnector - Network
Connector default-nc Started
2007-06-18 10:04:14,253 [main ] INFO TransportConnector - Connector
vm://shakira-slave Started
2007-06-18 10:04:14,281 [main ] INFO MasterConnector - Starting a
network connection between vm://shakira-slave#0 and tcp://null:0 has been
established.
2007-06-18 10:04:14,287 [main ] INFO BrokerService - ActiveMQ JMS
Message Broker (shakira-slave,
ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
2007-06-18 10:04:14,333 [Thread-4 ] INFO MasterConnector - Slave
connection between vm://shakira-slave#0 and
tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
2007-06-18 10:04:14,740 [ Agent Notifier] INFO NetworkConnector - Establishing
network connection between from vm://shakira-slave?network=true to
tcp://master.cac.washington.edu:61616
2007-06-18 10:04:14,762 [Thread-7 ] INFO DemandForwardingBridge -
Network connection between vm://shakira-slave#2 and
tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master) has been
established.
2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network
connection between vm://shakira-slave#0 and
tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel was inactive
for too long.
2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN BrokerService - Master Failed -
starting all connectors
2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to
startAllConnectors
The slave keeps timing out and then fails to startup the connectors. I
have no idea why the channel is inactive since there are thousands of
events per minute being sent into ActiveMQ topics using the a failover URL
of:
failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false
If I kill the master server (kill -9) the slave doesn't take over at this
point. The slave doesn't seem to log any messages once startAllConnectors fails,
although the activeMQ process is still running.
I've also tried restarting the slave once the slave's connection to the
master broker times out and the master server seems to think the slave is
still connected and won't let the slave reconnect.
My master config is:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-master" useJmx="true"
xmlns="http://activemq.org/config/1.0";>
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://master.cac.washington.edu:61616"
discoveryUri="multicast://default"/>
<transportConnector name="stomp"
uri="stomp://master.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
And my slave config is:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-slave" useJmx="true"
masterConnectorURI="tcp://master.cac.washington.edu:61616" shutdownOnMasterFailure="false"
xmlns="http://activemq.org/config/1.0";>
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://slave.cac.washington.edu:61616"
discoveryUri="multicast://default"/>
<transportConnector name="stomp"
uri="stomp://slave.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
Has anyone had the same problem? Our product is getting ready to be
released and the reliablility problems with the pure master/slave setup is
becoming a show stopper for us. Any help would be greatly appreciated.
Thanks,
Todd
