[ https://issues.apache.org/jira/browse/KAFKA-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366362#comment-14366362 ]
InduR commented on KAFKA-2028: ------------------------------ The Data and log files are also mapped to be on tmp; # A comma seperated list of directories under which to store log files log.dirs=/tmp/kafka-logs The server had not been rebooted in the last two months since kafka was first installed and started running. [root@lablx0025 config]# uptime 19:07:23 up 63 days, 3:19, 1 user, load average: 0.00, 0.00, 0.00 So to resolve this issue should I re point server.properties and ZK.Properties to a new file system and start the instances? Thanks! > Unable to start the ZK instance after myid file was missing and had to > recreate it. > ----------------------------------------------------------------------------------- > > Key: KAFKA-2028 > URL: https://issues.apache.org/jira/browse/KAFKA-2028 > Project: Kafka > Issue Type: Bug > Components: admin > Affects Versions: 0.8.1.1 > Environment: Non Prod > Reporter: InduR > > Created a Dev 3 node cluster environment in Jan and the environment has been > up and running without any issues until few days. > Kafka server stopped running but ZK listener was up .Noticed that the Myid > file was missing in all 3 servers. > Recreated the file when ZK was still running did not help. > Stopped all of the ZK /kafka server instances and see the following error > when starting ZK. > kafka_2.10-0.8.1.1 > OS : RHEL > [root@lablx0025 bin]# ./zookeeper-server-start.sh > ../config/zookeeper.properties & > [1] 31053 > [***** bin]# [2015-03-17 15:04:33,876] INFO Reading configuration from: > ../config/zookeeper.properties (org.apache.zookeeper. > > server.quorum.QuorumPeerConfig) > [2015-03-17 15:04:33,885] INFO Defaulting to majority quorums > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) > [2015-03-17 15:04:33,911] DEBUG preRegister called. > Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4891d863, > name=log4j:logger=kafka (k > afka) > [2015-03-17 15:04:33,915] INFO Starting quorum peer > (org.apache.zookeeper.server.quorum.QuorumPeerMain) > [2015-03-17 15:04:33,940] INFO binding to port 0.0.0.0/0.0.0.0:2181 > (org.apache.zookeeper.server.NIOServerCnxn) > [2015-03-17 15:04:33,966] INFO tickTime set to 3000 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [2015-03-17 15:04:33,966] INFO minSessionTimeout set to -1 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [2015-03-17 15:04:33,966] INFO maxSessionTimeout set to -1 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [2015-03-17 15:04:33,966] INFO initLimit set to 5 > (org.apache.zookeeper.server.quorum.QuorumPeer) > [2015-03-17 15:04:34,023] ERROR Failed to increment parent cversion for: > /consumers/console-consumer-6249/offsets/test (org.apache.zoo > > keeper.server.persistence.FileTxnSnapLog) > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /consumers/console-consumer-6249/offsets/test > at > org.apache.zookeeper.server.DataTree.incrementCversion(DataTree.java:1218) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:222) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:150) > at > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) > at > org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) > [2015-03-17 15:04:34,027] FATAL Unable to load database on disk > (org.apache.zookeeper.server.quorum.QuorumPeer) > java.io.IOException: Failed to process transaction type: 2 error: > KeeperErrorCode = NoNode for /consumers/console-consumer-6249/offset > > s/test > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152) > at > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) > at > org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) > [2015-03-17 15:04:34,027] FATAL Unexpected exception, exiting abnormally > (org.apache.zookeeper.server.quorum.QuorumPeerMain) > java.lang.RuntimeException: Unable to run quorum server > at > org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:401) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76) > Caused by: java.io.IOException: Failed to process transaction type: 2 error: > KeeperErrorCode = NoNode for /consumers/console-consumer- > > 6249/offsets/test > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152) > at > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222) > at > org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398) > ... 3 more > ***************** > Zookeeper properties: > # Licensed to the Apache Software Foundation (ASF) under one or more > # contributor license agreements. See the NOTICE file distributed with > # this work for additional information regarding copyright ownership. > # The ASF licenses this file to You under the Apache License, Version 2.0 > # (the "License"); you may not use this file except in compliance with > # the License. You may obtain a copy of the License at > # > # http://www.apache.org/licenses/LICENSE-2.0 > # > # Unless required by applicable law or agreed to in writing, software > # distributed under the License is distributed on an "AS IS" BASIS, > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > # See the License for the specific language governing permissions and > # limitations under the License. > # the directory where the snapshot is stored. > dataDir=/tmp/zookeeper > # the port at which the clients will connect > clientPort=2181 > # disable the per-ip limit on the number of connections since this is a > non-production config > #maxClientCnxns=0 > server.1=lablx0023:2888:3888 > server.2=lablx0024:2888:3888 > server.3=lablx0025:2888:3888 > #add here more servers if you want > initLimit=5 > syncLimit=2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)