Thank you for pointing that out! Since we only have one broker, I disabled the multicast discovery.
Unfortunately, that didn't fix the problem. The restarts are still happening. Sometimes they occur within few minutes of startup, sometimes takes 4-5 minutes. The only relevant info I see in "activemq.log" is this: 2009-07-30 17:42:54,652 [MQ ShutdownHook] INFO BrokerService - ActiveMQ Message Broker (ai_jms, ID:server2-53314-1248975478282-0:0) is shutting down 2009-07-30 17:42:54,653 [MQ ShutdownHook] DEBUG BrokerService - Caught exception, must be shutting down: java.lang.IllegalStateException: Shutdown in progress 2009-07-30 17:42:54,657 [MQ ShutdownHook] INFO NetworkConnector - Network Connector local-nc Stopped 2009-07-30 17:42:56,089 [31.140.18:56800] Which makes me thing that something is sending a kill command to the process, but I have no idea what that could be. _Gary rajdavies wrote: > > first - try disabling multicast discovery in the activemq configuration > > On 29 Jul 2009, at 22:17, GaryG wrote: > >> >> We've recently updated to ActiveMQ 5.2.0, and are running it as a >> standalone >> server, with a config file very similar to the basic example >> provided out of >> the box. There has been no code change in the Java code that talks >> to the >> AMQ server. Yet, once started up, the ActiveMQ server process is >> restarting >> itself every 5 minutes. >> >> Looking at the "activemq.log", I see the following: >> =============== >> . >> . >> . >> 2009-07-29 20:48:01,949 [31.140.17:41258] DEBUG AbstractRegion >> - Adding destination: topic:// >> ActiveMQ.Advisory.Producer.Topic.dr01b_wf1 >> 2009-07-29 20:48:01,951 [31.140.17:41258] DEBUG AbstractRegion >> - Adding destination: >> topic://ActiveMQ.Advisory.Producer.Topic.WorkerFarmControl >> 2009-07-29 20:48:01,952 [31.140.17:41258] DEBUG AbstractRegion >> - Adding destination: topic:// >> ActiveMQ.Advisory.Producer.Topic.dr01a_wf1 >> 2009-07-29 20:48:01,956 [31.140.17:41258] DEBUG AbstractRegion >> - Adding destination: topic://ControllerCCTopicSend >> 2009-07-29 20:48:01,957 [31.140.17:41258] DEBUG >> JournalPersistenceAdapter >> - Waking for checkpoint to complete. >> 2009-07-29 20:48:01,957 [eckpoint Worker] DEBUG >> JournalPersistenceAdapter >> - Checkpoint started. >> 2009-07-29 20:48:01,979 [eckpoint Worker] DEBUG >> JournalPersistenceAdapter >> - Checkpoint done. >> 2009-07-29 20:48:01,984 [31.140.17:41258] DEBUG AbstractRegion >> - Adding destination: >> topic://ActiveMQ.Advisory.Producer.Topic.ControllerCCTopicSend >> 2009-07-29 20:52:55,790 [MQ ShutdownHook] INFO BrokerService >> - ActiveMQ Message Broker (ai_jms, >> ID:server2-55452-1248900479410-0:0) is >> shutting down >> 2009-07-29 20:52:55,791 [MQ ShutdownHook] DEBUG BrokerService >> - Caught exception, must be shutting down: >> java.lang.IllegalStateException: >> Shutdown in progress >> 2009-07-29 20:52:55,796 [MQ ShutdownHook] INFO NetworkConnector >> - Network Connector local-nc Stopped >> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TransportConnection >> - Stopping connection: /1.1.1.18:56300 >> 2009-07-29 20:52:57,294 [31.140.18:56300] DEBUG TcpTransport >> - Stopping transport tcp:///1.1.1.18:56300 >> 2009-07-29 20:52:57,297 [31.140.18:56300] DEBUG TransportConnection >> - Stopped transport: /1.1.1.18:56300 >> 2009-07-29 20:52:57,298 [31.140.18:56300] DEBUG TransportConnection >> - Cleaning up connection resources: /1.1.1.18:56300 >> 2009-07-29 20:52:57,299 [31.140.18:56300] DEBUG >> JournalPersistenceAdapter >> - Waking for checkpoint to complete. >> 2009-07-29 20:52:57,299 [eckpoint Worker] DEBUG >> JournalPersistenceAdapter >> - Checkpoint started. >> 2009-07-29 20:52:57,331 [eckpoint Worker] DEBUG >> JournalPersistenceAdapter >> - Checkpoint done. >> 2009-07-29 20:52:57,338 [31.140.18:56300] DEBUG AbstractRegion >> - Removing consumer: ID:server2-49559-1248899931358-0:2:-1:1 >> . >> . >> . >> ------------- >> >> So you can see for a while, the broker's adding topics and such, and >> then >> all of a sudden it gets an "IllegalStateException", and just restarts. >> >> >> On the clients, when this happens we get this error: >> >> -------------- >> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm - >> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1, >> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:224 >> 2009-07-29 20:52:48,550 [ActiveMQ Session Task] DEBUG jms_comm - >> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to >> messageListener! >> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm - >> ActiveMQMessageReceiver[dr01b_wf1] got msg from = STG_AIC1, >> jmsMsgID=ID:server1-41646-1248899975358-0:2:6:1:225 >> 2009-07-29 20:52:49,896 [ActiveMQ Session Task] DEBUG jms_comm - >> ActiveMQMessageReceiver[dr01b_wf1] sent processed msg to >> messageListener! >> 2009-07-29 20:52:57,865 [ActiveMQ Connection Worker: >> tcp://server2/1.1.1.18:61616] ERROR jms_comm - FactoryUtil.onException >> javax.jms.JMSException: java.io.EOFException >> at >> org >> .apache >> .activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) >> at >> org >> .apache >> .activemq >> .ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773) >> at >> org >> .apache >> .activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1790) >> at >> org >> .apache >> .activemq.transport.TransportFilter.onException(TransportFilter.java: >> 99) >> at >> org >> .apache >> .activemq >> .transport.ResponseCorrelator.onException(ResponseCorrelator.java:114) >> at >> org >> .apache >> .activemq.transport.TransportFilter.onException(TransportFilter.java: >> 99) >> at >> org >> .apache >> .activemq.transport.TransportFilter.onException(TransportFilter.java: >> 99) >> at >> org >> .apache >> .activemq >> .transport >> .WireFormatNegotiator.onException(WireFormatNegotiator.java:152) >> at >> org >> .apache >> .activemq >> .transport.InactivityMonitor.onException(InactivityMonitor.java:244) >> at >> org >> .apache >> .activemq >> .transport.TransportSupport.onException(TransportSupport.java:96) >> at >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java: >> 189) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: java.io.EOFException >> at java.io.DataInputStream.readInt(DataInputStream.java:375) >> at >> org >> .apache >> .activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269) >> at >> org >> .apache >> .activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java: >> 210) >> at >> org >> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java: >> 202) >> at >> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java: >> 185) >> ... 1 more >> 2009-07-29 20:52:57,895 [ActiveMQ Connection Worker: >> tcp://server2.mss.iss.net/1.1.1.18:61616] DEBUG jms_comm - JMS >> Exception >> occured. Server is down. >> 2009-07-29 20:52:58,263 [Thread-27] ERROR jms_comm.log - >> ActiveMQMessageSender[ControllerTaskStatusTopic] Error sending >> message: >> javax.jms.JMSException: Channel was inactive for too long: >> server2/1.1.1.18:61616 >> -------------- >> >> >> Our "activemq.xml" file is pretty simple - it's almost exactly the >> default >> config with just some minor modifications: >> ---------------------- >> >> <!-- >> Licensed to the Apache Software Foundation (ASF) under one or more >> contributor license agreements. See the NOTICE file distributed >> with >> this work for additional information regarding copyright ownership. >> The ASF licenses this file to You under the Apache License, >> Version 2.0 >> (the "License"); you may not use this file except in compliance >> with >> the License. You may obtain a copy of the License at >> >> http://www.apache.org/licenses/LICENSE-2.0 >> >> Unless required by applicable law or agreed to in writing, software >> distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >> See the License for the specific language governing permissions and >> limitations under the License. >> --> >> <!-- START SNIPPET: example --> >> <beans >> xmlns="http://www.springframework.org/schema/beans"; >> xmlns:amq="http://activemq.apache.org/schema/core"; >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> http://activemq.apache.org/schema/core >> http://activemq.apache.org/schema/core/activemq-core.xsd >> http://activemq.apache.org/camel/schema/spring >> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> >> >> <!-- Allows us to use system properties as variables in this >> configuration >> file --> >> <bean >> class >> = >> "org >> .springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >> <property name="locations"> >> <value>file:///${activemq.base}/conf/credentials.properties</ >> value> >> </property> >> </bean> >> >> <broker xmlns="http://activemq.apache.org/schema/core"; >> brokerName="ai_jms" >> useJmx="true" dataDirectory="${activemq.base}/data"> >> >> <!-- The store and forward broker networks ActiveMQ will listen >> to --> >> <networkConnectors> >> <!-- by default just auto discover the other brokers --> >> <!--<networkConnector name="default-nc" uri="multicast:// >> default"/> >> --> >> <!-- Example of a static configuration: --> >> <networkConnector name="local-nc" >> uri="static://(tcp://localhost:61616)"/> >> </networkConnectors> >> >> <persistenceAdapter> >> <journaledJDBC journalLogFiles="10" >> dataDirectory="${activemq.base}/activemq-data" dataSource="#derby- >> ds"/> >> </persistenceAdapter> >> >> >> <!-- The maximum about of space the broker will use before >> slowing down >> producers --> >> <systemUsage> >> <systemUsage> >> <memoryUsage> >> <memoryUsage limit="512 mb"/> >> </memoryUsage> >> <storeUsage> >> <storeUsage limit="1 gb" name="foo"/> >> </storeUsage> >> <tempUsage> >> <tempUsage limit="100 mb"/> >> </tempUsage> >> </systemUsage> >> </systemUsage> >> >> <!-- The transport connectors ActiveMQ will listen to --> >> <transportConnectors> >> <transportConnector name="openwire" uri="tcp://localhost:61616" >> discoveryUri="multicast://default"/> >> </transportConnectors> >> >> </broker> >> >> <!-- An embedded servlet engine for serving up the Admin console --> >> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0";> >> <connectors> >> <nioConnector port="8161"/> >> </connectors> >> >> <handlers> >> <webAppContext contextPath="/admin" >> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/> >> <webAppContext contextPath="/demo" >> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/> >> <webAppContext contextPath="/fileserver" >> resourceBase="${activemq.base}/webapps/fileserver" >> logUrlOnStart="true"/> >> </handlers> >> </jetty> >> >> <!-- Embedded Derby DataSource Sample Setup --> >> <bean id="derby-ds" class="org.apache.derby.jdbc.EmbeddedDataSource"> >> <property name="databaseName" value="derbydb"/> >> <property name="createDatabase" value="create"/> >> </bean> >> >> </beans> >> <!-- END SNIPPET: example --> >> >> >> >> ---------------------- >> >> >> And our Java clients use the following URL to connect to the server: >> >> tcp://${jms.server}:${jms.port}? >> keepAlive >> = >> true >> & >> ;connectionTimeout >> = >> 0 >> & >> ;wireFormat >> .maxInactivityDuration >> =0&jms.useAsyncSend=true&jms.prefetchPolicy.all=250" >> >> >> Notice that I tried setting the connectionTimeout and >> maxInactivityDuration >> to "0" thinking that this would turn off inactivity checking. >> This didn't help. Further, the system does try to send through a >> LOT of >> traffic when we bring it up, so there's definitely no "inactivity". >> >> As I mentioned, the exact same codebase was working perfectly fine >> with >> ActiveMQ 4.1.1. There's been no code changes, and no changes in the >> nature >> of the logs we're sending through. The only change has been >> upgrading to >> AMQ 5.2.0 on the server side, and using the new jar file on the >> Java client side. >> >> Any help would be appreciated, as I'm totally stumped as to why the >> server >> keeps restarting itself. Our message content hasn't changed, >> so why would the EOF file errors or "channel inactive" errors be >> happening?? >> >> Thanks, >> Gary >> >> -- >> View this message in context: >> http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24727769.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > Rob Davies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ > > > > > > -- View this message in context: http://www.nabble.com/Standalone-server-restarting-itself-every-5-minutes---java.lang.IllegalStateException-tp24727769p24743651.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
