Hi I need urgent help in this matter. I hope anyone can help.
We are running Artemis 2.4.0 We have a Master/Slave Cluster Config. We have been running it since september 2018 in Prod. Twice we have had issues with failover (out of about 50-100 or so failovers). So failover mostly works. When it failed over today it posted a lot of errors and then just published DLQ and ExpiryQueue on the Slave. All other queues and 150 000 messages were lost. Broker.xml <?xml version='1.0'?> <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"> <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core "> <name>ActiveMQ1</name> <persistence-enabled>true</persistence-enabled> <journal-type>ASYNCIO</journal-type> <paging-directory>./data/paging</paging-directory> <bindings-directory>./data/bindings</bindings-directory> <journal-directory>./data/journal</journal-directory> <large-messages-directory>./data/large-messages</large-messages-directory> <journal-datasync>true</journal-datasync> <journal-min-files>2</journal-min-files> <journal-pool-files>-1</journal-pool-files> <disk-scan-period>5000</disk-scan-period> <max-disk-usage>100</max-disk-usage> <critical-analyzer>true</critical-analyzer> <critical-analyzer-timeout>120000</critical-analyzer-timeout> <critical-analyzer-check-period>60000</critical-analyzer-check-period> <critical-analyzer-policy>HALT</critical-analyzer-policy> <acceptors> <acceptor name="artemis">tcp://nodep01:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpol l=true;amqpCredits=1000;amqpLowCredits=300</acceptor> <acceptor name="amqp">tcp://nodep01:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredit s=300</acceptor> <acceptor name="stomp">tcp://nodep01:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor> <acceptor name="hornetq">tcp://nodep01:5445?protocols=HORNETQ,STOMP;useEpoll=true</acceptor> <acceptor name="artemis+ssl">tcp://nodep01:61443?sslEnabled=true;keyStorePath=/opt/brokers/ActiveMQServer1/activemq.keystore;keyStorePassword=CCCC;tcpSen dBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor> </acceptors> <cluster-user>aaaaaa</cluster-user> <cluster-password>bbbbbbb</cluster-password> <cluster-connections> <cluster-connection name="ActiveMQClusterPROD"> <connector-ref>artemis</connector-ref> <min-large-message-size>33554432</min-large-message-size> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <static-connectors> <connector-ref>ActiveMQ2-Connector</connector-ref> </static-connectors> </cluster-connection> </cluster-connections> <ha-policy> <replication> <master> <check-for-live-server>true</check-for-live-server> </master> </replication> </ha-policy> <security-settings> <security-setting match="#"> <permission type="createNonDurableQueue" roles="amq"/> <permission type="deleteNonDurableQueue" roles="amq"/> <permission type="createDurableQueue" roles="amq"/> <permission type="deleteDurableQueue" roles="amq"/> <permission type="createAddress" roles="amq"/> <permission type="deleteAddress" roles="amq"/> <permission type="consume" roles="amq"/> <permission type="browse" roles="amq"/> <permission type="send" roles="amq"/> <permission type="manage" roles="amq"/> </security-setting> </security-settings> <address-settings> <address-setting match="activemq.management#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-create-jms-queues>true</auto-create-jms-queues> <auto-create-jms-topics>true</auto-create-jms-topics> </address-setting> <address-setting match="#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-create-jms-queues>true</auto-create-jms-queues> <auto-create-jms-topics>true</auto-create-jms-topics> </address-setting> </address-settings> <addresses> <address name="DLQ"> <anycast> <queue name="DLQ" /> </anycast> </address> <address name="ExpiryQueue"> <anycast> <queue name="ExpiryQueue" /> </anycast> </address> </addresses> </core> </configuration> *Any ideas what happend here? Can messages and queues be restored?* *Master:* A few of these (There were more): 16:32:15,845 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session dc5b6ac7-4bd0-11e9-9db5- 005056926b6e 16:32:15,847 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session dc5b6ac7-4bd0-11e9-9db5-005056926b6e 16:32:15,847 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17:1:-1 16:32:15,847 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-17 :1:-1 16:32:16,002 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 16:32:16,003 INFO [io.hawt.HawtioContextListener] Destroying hawtio services 16:32:16,088 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin 16:32:16,093 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin 16:32:16,104 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056 926b6e] stopped, uptime 4 days 3 hours *Meanwhile on Slave:* 16:32:15,867 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote 16:32:15,868 INFO [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results. 16:32:15,869 INFO [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes. 16:32:15,869 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results. 16:32:15,889 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 16:32:15,893 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 16:32:15,944 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] 16:32:15,999 INFO [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=06926557-2906-11e8-a15f-005056926b6e to become 'live' 16:32:16,450 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server. 16:32:34,825 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue DLQ on address DLQ 16:32:34,825 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying queue ExpiryQueue on address ExpiryQueue 16:32:35,156 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live 16:32:35,171 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:61616 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE] 16:32:35,174 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:5445 for protocols [HORNETQ,STOMP] 16:32:35,175 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:61443 for protocols [CORE,MQTT,AMQP,STOMP,HORNETQ,OPENWIRE] 16:32:35,177 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:5672 for protocols [AMQP] 16:32:35,179 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:1883 for protocols [MQTT] 16:32:35,181 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at nodep02:61613 for protocols [STOMP] 16:33:51,161 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:53781 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:34:10,061 INFO [io.hawt.web.LoginServlet] hawtio login is using 1800 sec. HttpSession timeout 16:34:11,891 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:34:52,912 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:35:01,183 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:47171 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:36:01,191 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:54700 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:36:12,705 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:36:53,104 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:37:11,203 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:16751 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:38:11,209 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20634 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:38:12,182 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:38:56,926 INFO [io.hawt.web.RBACMBeanInvoker] Instance not found: org.apache.activemq.artemis:broker="ActiveMQ1",component=addresses,address="DLQ" 16:39:21,216 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:61541 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:40:21,225 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:49708 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:41:18,720 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) while handshaking has occurred. 16:41:31,234 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:6685 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:42:31,240 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:10050 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:43:41,254 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:38611 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:44:41,264 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:8729 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:45:51,271 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119014: Did not receive data from /aaa.bbb.ccc.ddd:20748 within the 60,000ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 16:46:32,151 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-167.amq (size=10,485,760) to replica. 16:46:33,100 INFO [org.apache.activemq.artemis.core.server] AMQ221025: Replication: sending AIOSequentialFile:/opt/brokers/ActiveMQServer2/./data/journal/activemq-data-168.amq (size=10,485,760) to replica. 16:46:33,170 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 3a2b4ddd-4f13-11e9-b0fb-00505692a0af 16:46:33,170 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 3a2b4ddd-4f13-11e9-b0fb-00505692a0af 16:46:33,170 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-13496-1552451286921-29:1:-1 16:46:33,171 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-13496-1552451286921-29:1:-1 16:46:33,178 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-13496-1552451286921-29:1:1 16:46:33,178 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-13496-1552451286921-29:1:1 16:46:33,182 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 4efbabd8-4f13-11e9-b0fb-00505692a0af 16:46:33,182 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 4efbabd8-4f13-11e9-b0fb-00505692a0af 16:46:33,182 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1 16:46:33,188 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session ID:xxx.yyy.zzz-30305-1552451287008-19:1:-1 16:46:33,199 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380d4fc6-4f13-11e9-b0fb-00505692a0af 16:46:33,199 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 380d4fc6-4f13-11e9-b0fb-00505692a0af 16:46:33,199 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380e1317-4f13-11e9-b0fb-00505692a0af 16:46:33,199 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 380e1317-4f13-11e9-b0fb-00505692a0af 16:46:33,201 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 384f3ac9-4f13-11e9-b0fb-00505692a0af 16:46:33,201 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 384f3ac9-4f13-11e9-b0fb-00505692a0af 16:46:33,201 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 384ffe1a-4f13-11e9-b0fb-00505692a0af 16:46:33,202 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 384ffe1a-4f13-11e9-b0fb-00505692a0af 16:46:33,210 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380a1b75-4f13-11e9-b0fb-00505692a0af 16:46:33,210 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 380a1b75-4f13-11e9-b0fb-00505692a0af 16:46:33,210 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 380f72a8-4f13-11e9-b0fb-00505692a0af 16:46:33,210 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 380f72a8-4f13-11e9-b0fb-00505692a0af 16:46:33,211 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 3852e44b-4f13-11e9-b0fb-00505692a0af 16:46:33,211 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 3852e44b-4f13-11e9-b0fb-00505692a0af 16:46:33,211 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 3853ceae-4f13-11e9-b0fb-00505692a0af 16:46:33,212 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 3853ceae-4f13-11e9-b0fb-00505692a0af 16:46:33,214 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 385b96df-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 385b96df-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 385c8140-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 385c8140-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 38629bc1-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 38629bc1-4f13-11e9-b0fb-00505692a0af 16:46:33,215 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 38638622-4f13-11e9-b0fb-00505692a0af 16:46:33,216 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 38638622-4f13-11e9-b0fb-00505692a0af 16:46:33,288 INFO [io.hawt.web.AuthenticationFilter] Destroying hawtio authentication filter 16:46:33,290 INFO [io.hawt.HawtioContextListener] Destroying hawtio services 16:46:33,391 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin 16:46:33,397 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin 16:46:33,408 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.4.0 [06926557-2906-11e8-a15f-005056926b6e] stopped, uptime 4 days 3 hours 16:46:37,163 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 16:46:37,266 INFO [org.apache.activemq.artemis.core.server] AMQ221000: backup Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) 16:46:37,288 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.45 16:46:37,292 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/bindings to /opt/brokers/ActiveMQServer2/./data/bindings/oldreplica.47 16:46:37,295 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.49 16:46:37,597 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/journal to /opt/brokers/ActiveMQServer2/./data/journal/oldreplica.51 16:46:37,628 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.45 16:46:37,852 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 16:46:38,056 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer2/./data/paging to /opt/brokers/ActiveMQServer2/./data/paging/oldreplica.47 16:46:38,091 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 16:46:38,157 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 2,147,483,648 16:46:38,252 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 16:46:38,252 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 16:46:38,253 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 16:46:38,254 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 16:46:38,328 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 16:46:42,502 INFO [io.hawt.HawtioContextListener] Initialising hawtio services 16:46:42,519 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties 16:46:42,521 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-) 16:46:42,524 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/brokers/ActiveMQServer2/tmp/uploads 16:46:42,540 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq,Litium,Enova" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" 16:46:42,572 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/brokers/ActiveMQServer2//etc/jolokia-access.xml] 16:46:42,617 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access control 16:46:42,788 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, 10.4.21.50, nodep02.lindex.local] 16:46:43,202 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://nodep02:8161 16:46:43,202 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://nodep02:8161/console/jolokia And then the Master were waking up again Back on the Master 16:46:27,913 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 16:46:27,999 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=t rue,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) 16:46:28,440 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/Activ eMQServer1/./data/bindings/oldreplica.18 16:46:28,447 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/bindings to /opt/brokers/A ctiveMQServer1/./data/bindings/oldreplica.20 16:46:28,450 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/Activ eMQServer1/./data/journal/oldreplica.18 16:46:28,478 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/journal to /opt/brokers/Ac tiveMQServer1/./data/journal/oldreplica.20 16:46:28,482 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/brokers/Activ eMQServer1/./data/paging/oldreplica.18 16:46:28,517 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/brokers/ActiveMQServer1/./data/paging to /opt/brokers/Act iveMQServer1/./data/paging/oldreplica.20 16:46:28,580 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 16:46:28,625 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined a s 2,147,483,648 16:46:28,675 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 16:46:28,675 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 16:46:28,675 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORN ETQ 16:46:28,676 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 16:46:28,676 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPE NWIRE 16:46:28,676 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 16:46:28,772 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.4.0 [null] started, waiting live to fail before it gets active 16:46:28,943 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 16:46:29,082 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 16:46:31,036 INFO [io.hawt.HawtioContextListener] Initialising hawtio services 16:46:31,053 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties 16:46:31,055 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to hawtio 1.5.5 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-) 16:46:31,057 INFO [io.hawt.jmx.UploadManager] Using file upload directory: /opt/brokers/ActiveMQServer1/tmp/uploads 16:46:31,072 INFO [io.hawt.web.AuthenticationFilter] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq,Litiumi,Enova" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" 16:46:31,101 INFO [io.hawt.web.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/opt/brokers/ActiveMQServer1//et c/jolokia-access.xml] 16:46:31,146 INFO [io.hawt.web.RBACMBeanInvoker] Using MBean [hawtio:type=security,area=jmx,rank=0,name=HawtioDummyJMXSecurity] for role based access contr ol 16:46:31,301 INFO [io.hawt.system.ProxyWhitelist] Initial proxy whitelist: [localhost, 127.0.0.1, ggg.hhh.jjj.49, nodep01.domain.local] 16:46:31,651 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://nodep01:8161 16:46:31,651 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://nodep01:8161/console/jolokia 16:46:31,651 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://nodep01:8161/console 16:46:33,206 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote 16:46:33,207 INFO [org.apache.activemq.artemis.core.server] AMQ221067: Waiting 30 seconds for quorum vote results. 16:46:33,207 INFO [org.apache.activemq.artemis.core.server] AMQ221068: Received all quorum votes. 16:46:33,207 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results. 16:46:33,209 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected bec ause of server shutdown [code=DISCONNECTED] 16:46:33,266 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected bec ause of server shutdown [code=DISCONNECTED] 16:46:33,273 WARN [org.apache.activemq.artemis.journal] File not opened, file=null: java.lang.NullPointerException: File not opened, file=null at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.checkOpened(AIOSequentialFile.java:328) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:242) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,274 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,274 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,275 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,275 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,276 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,279 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,279 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,280 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,280 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,281 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,281 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,282 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,282 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,283 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,283 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,284 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,284 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,285 WARN [org.apache.activemq.artemis.journal] AMQ142027: Error on writing data! File not opened, file=null code - -1: java.lang.Exception: File n ot opened, file=null at org.apache.activemq.artemis.core.io.DummyCallback.onError(DummyCallback.java:36) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$DelegateCallback.onError(AbstractSequentialFile.java:280) [artemis-journal-2.4.0.jar:2 .4.0] at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.writeDirect(AIOSequentialFile.java:245) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFile$LocalBufferObserver.flushBuffer(AbstractSequentialFile.java:306) [artemis-journal-2.4. 0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flushBatch(TimedBuffer.java:310) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.flush(TimedBuffer.java:281) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.io.AbstractSequentialFileFactory.flush(AbstractSequentialFileFactory.java:195) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.flush(JournalImpl.java:2194) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.stop(JournalImpl.java:2356) [artemis-journal-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:266) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.stop(JournalStorageManager.java:203) [artemis-server-2.4.0.jar:2. 4.0] at org.apache.activemq.artemis.core.replication.ReplicationEndpoint.stop(ReplicationEndpoint.java:327) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stopComponent(ActiveMQServerImpl.java:1256) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:259) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,318 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE mes sage=AMQ119026: Backup Server was not yet in sync with live] at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:314) [artemis-server-2.4.0.jar: 2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2951) [artemis-server-2.4.0.jar:2.4. 0] 16:46:33,995 INFO [io.hawt.web.keycloak.KeycloakServlet] Keycloak integration is disabled 16:46:40,486 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 16:46:50,033 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 16:47:00,034 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 16:47:10,022 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped 16:47:20,026 WARN [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html