Hi all,
I’m running a stand-alone camel app with an embedded activemq broker and am
running into unclean shutdowns. When shutting down, I get a graceful shutdown
of the camel context, and then an InterruptedExcpetion from KahaDB when
ActiveMQ attempts shutdown afterwards.
[ Thread-6] MainSupport INFO Apache
Camel 2.14.2 stopping
[ Thread-6] SpringCamelContext INFO Apache
Camel 2.14.2 (CamelContext: camel) is shutting down
[ Thread-6] DefaultShutdownStrategy INFO Starting
to graceful shutdown 2 routes (timeout 300 seconds)
[amel) thread #2 - ShutdownTask] DefaultShutdownStrategy INFO Route:
route2 shutdown complete, was consuming from:
Endpoint[file:///home/vagrant/fedora3data/objectStore?noop=true&recursive=true&sendEmptyMessageWhenIdle=true]
[amel) thread #2 - ShutdownTask] DefaultShutdownStrategy INFO Route:
route1 shutdown complete, was consuming from: Endpoint[activemq://foxml]
[ Thread-6] DefaultShutdownStrategy INFO Graceful
shutdown of 2 routes completed in 0 seconds
[ Thread-6] SpringCamelContext INFO Apache
Camel 2.14.2 (CamelContext: camel) uptime 3.128 seconds
[ Thread-6] SpringCamelContext INFO Apache
Camel 2.14.2 (CamelContext: camel) is shutdown in 0.494 seconds
[ Thread-6] BrokerService INFO Apache
ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) is shutting
down
[ Thread-6] TransportConnector INFO Connector
vm stopped
[ Thread-6] PListStoreImpl INFO
PListStore:[/home/vagrant/spring/migration-utils/src/data/myBroker/tmp_storage]
stopped
[ Thread-6] KahaDBStore INFO Stopping
async queue tasks
[ Thread-6] KahaDBStore ERROR Could not
stop service:
KahaDB:[/home/vagrant/spring/migration-utils/src/data/myBroker/KahaDB]. Reason:
java.lang.InterruptedException
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:588)
at
org.apache.activemq.store.kahadb.KahaDBStore.doStop(KahaDBStore.java:247)
at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)
at
org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStop(KahaDBPersistenceAdapter.java:250)
at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:792)
at
org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)
at
org.apache.activemq.xbean.XBeanBrokerService.destroy(XBeanBrokerService.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:350)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:538)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:514)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:831)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:483)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1092)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1066)
at
org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1012)
at org.apache.camel.util.IOHelper.close(IOHelper.java:326)
at org.apache.camel.util.IOHelper.close(IOHelper.java:390)
at org.apache.camel.spring.Main.doStop(Main.java:183)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at
org.fcrepo.migration.processors.ShutdownProcessor$1.run(ShutdownProcessor.java:23)
The shutdown does eventually continue and complete after hanging for a bit,
with the following message:
[ Thread-6] BrokerService INFO Apache
ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) uptime 4.065
seconds
[ Thread-6] BrokerService INFO Apache
ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) is shutdown
[ Thread-6] DisposableBeanAdapter WARN
Invocation of destroy method 'destroy' failed on bean with name 'broker':
java.lang.InterruptedException
[WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] was
interrupted but is still alive after waiting at least 14995msecs
[WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] will
linger despite being asked to die via interruption
[WARNING] thread
Thread[ConcurrentQueueStoreAndDispatch,5,org.apache.camel.spring.Main] will
linger despite being asked to die via interruption
[WARNING] NOTE: 2 thread(s) did not finish despite being asked to via
interruption. This is not a problem with exec:java, it is a problem with the
running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup
org.apache.camel.maven.RunMojo$IsolatedThreadGroup[name=org.apache.camel.spring.Main,maxpri=10]
java.lang.IllegalThreadStateException
The interrupted exception makes sense. I’m just wondering what’s the best
approach to handling this properly? I’m shutting down the application using
another thread as per
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html. Is the
problem perhaps how I’m shutting down the Spring context by calling stop on
Main? Is there a better way to stop a command-line camel app to ensure things
get handled in order and wait for the broker and persistence services to
shutdown cleanly?
I’ve personally tried shutting down the camel context manually before the
spring context, as well as shutdown hooks on the broker bean. Nothing seems to
be working. Here’s my shutdown code:
package org.fcrepo.migration.processors;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.spring.Main;
/**
* Shuts down the application.
*
* @author Daniel Lamb
*/
public class ShutdownProcessor implements Processor {
private Thread stop;
@Override
public void process(final Exchange exchange) throws Exception {
if (stop == null) {
stop = new Thread() {
@Override
public void run() {
try {
Main.getInstance().stop();
}
catch (Exception e) {
// ignore
}
}
};
}
stop.start();
}
}
Here’s my configs (pretty much ripped straight out of the camel activemq-tomcat
example:
Camel:
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:p="http://www.springframework.org/schema/p";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd";>
<!-- config -->
<bean id="bridgePropertyPlaceholder"
class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="location" value="classpath:migration.properties"/>
</bean>
<!-- activemq -->
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL"
value="vm://myBroker?create=false&waitForStart=5000"/>
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring";
id="camel"
shutdownRunningTask="CompleteAllTasks">
<package>org.fcrepo.migration.routes</package>
</camelContext>
</beans>
Activemq:
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:context="http://www.springframework.org/schema/context";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<broker xmlns="http://activemq.apache.org/schema/core";
id="broker"
brokerName="myBroker"
useShutdownHook=“false"
persistent="true"
dataDirectory="${activemqData}">
<transportConnectors>
<transportConnector name="vm" uri="vm://myBroker"/>
</transportConnectors>
</broker>
</beans>
Any advice/experience would be greatly appreciated.
~Danny
