LevelDB support has been deprecated for many years now and is going to be removed in the next version of ActiveMQ (5.17.0) so I would suggest trying to migrate over to KahaDB.
On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <venkitaswa...@gmail.com> wrote: > Hello > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. It's > a very old system. However last night we started to get these errors and > the broker did not come up. Any ideas on how to fix this would be > appreciated. I can provide further log messages as needed, if sufficient > direction is provided. Thank you! > > Here are the logs > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > is starting | > org.apache.activemq.broker.BrokerService | main > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat > .maxFrameSize=104857600 | > org.apache.activemq.transport.TransportServerThreadSupport | main > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > org.apache.activemq.broker.TransportConnector | main > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat > at > > org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] > at > > org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] > at > > org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] > at > > org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] > at > > org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] > at > > org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_232] > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] > at > > org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] > at > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] > 2020-08-07 02:44:46,669 | INFO | Refreshing > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date [Fri > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,10.184.11.9:2181, > 10.184.11.7:2181//activemq/ap > ps/mkt/lpf/prod-usc1a/BROKER01P] | org.apache.activemq.broker.BrokerService > | main > 2020-08-07 02:44:51,492 | INFO | Client > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT > | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,494 | INFO | Client > environment:host.name > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal > | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,496 | INFO | Client environment:java.version=1.8.0_232 > | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle > Corporation | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,500 | INFO | Client > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre > | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,504 | INFO | Client > > environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar > | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,505 | INFO | Client > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > | org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,506 | INFO | Client > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,509 | INFO | Client > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,510 | INFO | Client > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper | > main > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 > sessionTimeout=2000 watcher=org.apache.activem > q.leveldb.replicated.groups.ZKClient@36060e | > org.apache.zookeeper.ZooKeeper | main > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state > change dispatcher thread > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: > javax.security.auth.login.LoginException: No JAAS configuration section > named 'Client' was found in specified JAAS > configuration file: > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue > connection to Zookeeper server without SASL authentication, if Zookeeper > server a > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( > 10.184.11.9:2181) > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | > main-SendThread(10.184.11.9:2181) > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed | > org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread > 2020-08-07 02:44:51,606 | INFO | Socket connection established to > 10.184.11.9/10.184.11.9:2181, initiating session | > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. > 11.9:2181) > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on server > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated > timeout = 4000 | org.apache. > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) > 2020-08-07 02:44:51,946 | INFO | Promoted to master | > org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to process > the events | > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > main-EventThread > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to process > the events | > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > main-EventThread > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ > BrokerService[BROKER01P] Task-1 > 2020-08-07 02:44:54,349 | INFO | Master started: > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ > BrokerService[BROKER01P] Task-2 > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > is starting | org.apache.activemq.broker.BrokerService | main > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > | org.apache.activemq.transport.TransportServerThreadSupport | main > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > org.apache.activemq.broker.TransportConnector | main > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true > | org.apache.activemq.transport.TransportServerThreadSupport | main > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | > org.apache.activemq.broker.TransportConnector | main > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > started | org.apache.activemq.broker.BrokerService | main > 2020-08-07 02:44:54,827 | INFO | For help or more information please see: > http://activemq.apache.org | org.apache.activemq.broker.BrokerService | > main > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb only > has 42345 mb of usable space. - resetting to maximum available disk space: > 42345 mb | org.apache.activemq.broker.BrokerService | main > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb > (current store usage is 25900 mb). The data directory: > /var/lib/activemq/data only has 42345 mb of usable space. - resetting to > maximum available disk space: 42345 mb | > org.apache.activemq.broker.BrokerService | main > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due to > exception, java.io.EOFException: File > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 | > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException > handler. > java.io.EOFException: File > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.activemq.console.Main.runTaskClass(Main.java:262) > at org.apache.activemq.console.Main.main(Main.java:115) >
