We've recently updated to ActiveMQ 5.2.0, and are running it as a standalone
server, with a config file very similar to the basic example provided out of
the box. There has been no code change in the Java code that talks to the
AMQ server. Yet, once started up, the ActiveMQ server process is restarting
itself every 5 minutes.
Looking at the "activemq.log", I see the following:
===============
.
.
.
2009-07-29 20:48:01,949 [31.140.17:41258] DEBUG AbstractRegion
- Adding destination: topic://ActiveMQ.Advisory.Producer.Topic.dr01b_wf1
2009-07-29 20:48:01,951 [31.140.17:41258] DEBUG AbstractRegion
- Adding destination:
topic://ActiveMQ.Advisory.Producer.Topic.WorkerFarmControl
2009-07-29 20:48:01,952 [31.140.17:41258] DEBUG AbstractRegion
- Adding destination: topic://ActiveMQ.Advisory.Producer.Topic.dr01a_wf1
2009-07-29 20:48:01,956 [31.140.17:41258] DEBUG AbstractRegion
- Adding destination: topic://ControllerCCTopicSend
2009-07-29 20:48:01,957 [31.140.17:41258] DEBUG JournalPersistenceAdapter
- Waking for checkpoint to complete.
2009-07-29 20:48:01,957 [eckpoint Worker] DEBUG JournalPersistenceAdapter
- Checkpoint started.
2009-07-29 20:48:01,979 [eckpoint Worker] DEBUG JournalPersistenceAdapter
- Checkpoint done.
2009-07-29 20:48:01,984 [31.140.17:41258] DEBUG AbstractRegion
- Adding destination:
topic://ActiveMQ.Advisory.Producer.Topic.ControllerCCTopicSend
2009-07-29 20:52:55,790 [MQ ShutdownHook] INFO BrokerService
- ActiveMQ Message Broker (ai_jms, ID:server2-55452-1248900479410-0:0) is
shutting down
2009-07-29 20:52:55,791 [MQ ShutdownHook] DEBUG BrokerService
- Caught exception, must be shutting down: java.lang.IllegalStateException:
Shutdown in progress
2009-07-29 20:52:55,796 [MQ ShutdownHook] INFO NetworkConnector
- Network Connector local-nc Stopped
2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TransportConnection
- Stopping connection: /1.1.1.18:56300
2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TcpTransport
- Stopping transport tcp:///1.1.1.18:56300
2009-07-29 20:52:57,297 [31.140.18:56300] DEBUG TransportConnection
- Stopped transport: /1.1.1.18:56300
2009-07-29 20:52:57,298 [31.140.18:56300] DEBUG TransportConnection
- Cleaning up connection resources: /1.1.1.18:56300
2009-07-29 20:52:57,299 [31.140.18:56300] DEBUG JournalPersistenceAdapter
- Waking for checkpoint to complete.
2009-07-29 20:52:57,299 [eckpoint Worker] DEBUG JournalPersistenceAdapter
- Checkpoint started.
2009-07-29 20:52:57,331 [eckpoint Worker] DEBUG JournalPersistenceAdapter
- Checkpoint done.
2009-07-29 20:52:57,338 [31.140.18:56300] DEBUG AbstractRegion
- Removing consumer: ID:server2-49559-1248899931358-0:2:-1:1
.
.
.
-------------
So you can see for a while, the broker's adding topics and such, and then
all of a sudden it gets an "IllegalStateException", and just restarts.
On the clients, when this happens we get this error:
--------------
2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:224
2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm -
ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to messageListener!
2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1,
jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:225
2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm -
ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to messageListener!
2009-07-29 20:52:57,865 [ActiveMQ Connection Worker:
tcp://server2/1.1.1.18:61616] ERROR jms_comm - FactoryUtil.onException
javax.jms.JMSException: java.io.EOFException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773)
at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1790)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:114)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:152)
at
org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:244)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:189)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
... 1 more
2009-07-29 20:52:57,895 [ActiveMQ Connection Worker:
tcp://server2.mss.iss.net/1.1.1.18:61616] DEBUG jms_comm - JMS Exception
occured. Server is down.
2009-07-29 20:52:58,263 [Thread-27] ERROR jms_comm.log -
ActiveMQMessageSender[ControllerTaskStatusTopic] Error sending message:
javax.jms.JMSException: Channel was inactive for too long:
server2/1.1.1.18:61616
--------------
Our "activemq.xml" file is pretty simple - it's almost exactly the default
config with just some minor modifications:
----------------------
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";>
<!-- Allows us to use system properties as variables in this configuration
file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:///${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="ai_jms"
useJmx="true" dataDirectory="${activemq.base}/data">
<!-- The store and forward broker networks ActiveMQ will listen to -->
<networkConnectors>
<!-- by default just auto discover the other brokers -->
<!--<networkConnector name="default-nc" uri="multicast://default"/>
-->
<!-- Example of a static configuration: -->
<networkConnector name="local-nc"
uri="static://(tcp://localhost:61616)"/>
</networkConnectors>
<persistenceAdapter>
<journaledJDBC journalLogFiles="10"
dataDirectory="${activemq.base}/activemq-data" dataSource="#derby-ds"/>
</persistenceAdapter>
<!-- The maximum about of space the broker will use before slowing down
producers -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="512 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
</transportConnectors>
</broker>
<!-- An embedded servlet engine for serving up the Admin console -->
<jetty xmlns="http://mortbay.com/schemas/jetty/1.0";>
<connectors>
<nioConnector port="8161"/>
</connectors>
<handlers>
<webAppContext contextPath="/admin"
resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
<webAppContext contextPath="/demo"
resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/>
<webAppContext contextPath="/fileserver"
resourceBase="${activemq.base}/webapps/fileserver" logUrlOnStart="true"/>
</handlers>
</jetty>
<!-- Embedded Derby DataSource Sample Setup -->
<bean id="derby-ds" class="org.apache.derby.jdbc.EmbeddedDataSource">
<property name="databaseName" value="derbydb"/>
<property name="createDatabase" value="create"/>
</bean>
</beans>
<!-- END SNIPPET: example -->
----------------------
And our Java clients use the following URL to connect to the server:
tcp://${jms.server}:${jms.port}?keepAlive=true&connectionTimeout=0&wireFormat.maxInactivityDuration=0&jms.useAsyncSend=true&jms.prefetchPolicy.all=250"
Notice that I tried setting the connectionTimeout and maxInactivityDuration
to "0" thinking that this would turn off inactivity checking.
This didn't help. Further, the system does try to send through a LOT of
traffic when we bring it up, so there's definitely no "inactivity".
As I mentioned, the exact same codebase was working perfectly fine with
ActiveMQ 4.1.1. There's been no code changes, and no changes in the nature
of the logs we're sending through. The only change has been upgrading to
AMQ 5.2.0 on the server side, and using the new jar file on the
Java client side.
Any help would be appreciated, as I'm totally stumped as to why the server
keeps restarting itself. Our message content hasn't changed,
so why would the EOF file errors or "channel inactive" errors be happening??
Thanks,
Gary
--
View this message in context:
http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24727769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
