InduR created KAFKA-2028: ---------------------------- Summary: Unable to start the ZK instance after myid file was missing and had to recreate it. Key: KAFKA-2028 URL: https://issues.apache.org/jira/browse/KAFKA-2028 Project: Kafka Issue Type: Bug Components: admin Affects Versions: 0.8.1.1 Environment: Non Prod Reporter: InduR
Created a Dev 3 node cluster environment in Jan and the environment has been up and running without any issues until few days. Kafka server stopped running but ZK listener was up .Noticed that the Myid file was missing in all 3 servers. Recreated the file when ZK was still running did not help. Stopped all of the ZK /kafka server instances and see the following error when starting ZK. kafka_2.10-0.8.1.1 OS : RHEL [root@lablx0025 bin]# ./zookeeper-server-start.sh ../config/zookeeper.properties & [1] 31053 [***** bin]# [2015-03-17 15:04:33,876] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper. server.quorum.QuorumPeerConfig) [2015-03-17 15:04:33,885] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2015-03-17 15:04:33,911] DEBUG preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4891d863, name=log4j:logger=kafka (k afka) [2015-03-17 15:04:33,915] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2015-03-17 15:04:33,940] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxn) [2015-03-17 15:04:33,966] INFO tickTime set to 3000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2015-03-17 15:04:33,966] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.quorum.QuorumPeer) [2015-03-17 15:04:33,966] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.quorum.QuorumPeer) [2015-03-17 15:04:33,966] INFO initLimit set to 5 (org.apache.zookeeper.server.quorum.QuorumPeer) [2015-03-17 15:04:34,023] ERROR Failed to increment parent cversion for: /consumers/console-consumer-6249/offsets/test (org.apache.zoo keeper.server.persistence.FileTxnSnapLog) org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/console-consumer-6249/offsets/test at org.apache.zookeeper.server.DataTree.incrementCversion(DataTree.java:1218) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:222) at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:150) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) [2015-03-17 15:04:34,027] FATAL Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer) java.io.IOException: Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for /consumers/console-consumer-6249/offset s/test at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) [2015-03-17 15:04:34,027] FATAL Unexpected exception, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain) java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:401) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) Caused by: java.io.IOException: Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for /consumers/console-consumer- 6249/offsets/test at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) ... 3 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)