That error means the consumer has never checkpointed any offset in ZK.
Thanks,
Jun
On Thu, Nov 21, 2013 at 10:31 PM, Tarang Dawer <tarang.da...@gmail.com>wrote:
> Hi Jun
>
> I tried the ConsumerOffsetChecker script for checking the offsets
>
> but there also , got an exception , with no information regarding the
> offset. Although when i start the consumer for the second time, it consumed
> the data, but still same response from the offset checker script.
>
> Please find the debug- level trace of the console output below.
>
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker \--group
> consumerGroup675437777 \--zkconnect 192.168.145.144:2181 \--topic
> test-topic;
> [2013-11-22 11:55:52,098] DEBUG Creating new ZookKeeper instance to connect
> to 192.168.145.144:2181. (org.I0Itec.zkclient.ZkConnection)
> [2013-11-22 11:55:52,098] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2013-11-22 11:55:52,103] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:host.name
> =hadoop-namenode
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:java.version=1.6.0_33
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:java.vendor=Sun
> Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client
> environment:java.home=/usr/java/jdk1.6.0_33/jre
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client
>
> environment:java.class.path=:bin/../core/target/scala-2.8.0/*.jar:bin/../perf/target/scala-2.8.0/kafka*.jar:bin/../libs/jopt-simple-3.2.jar:bin/../libs/log4j-1.2.15.jar:bin/../libs/metrics-annotation-2.2.0.jar:bin/../libs/metrics-core-2.2.0.jar:bin/../libs/scala-compiler.jar:bin/../libs/scala-library.jar:bin/../libs/slf4j-api-1.7.2.jar:bin/../libs/slf4j-simple-1.6.4.jar:bin/../libs/snappy-java-1.0.4.1.jar:bin/../libs/zkclient-0.3.jar:bin/../libs/zookeeper-3.3.4.jar:bin/../kafka_2.8.0-0.8.0-beta1.jar
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client
>
> environment:java.library.path=/usr/java/jdk1.6.0_33/jre/lib/amd64/server:/usr/java/jdk1.6.0_33/jre/lib/amd64:/usr/java/jdk1.6.0_33/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client
> environment:os.version=2.6.32-279.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:user.name=impadmin
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client environment:user.home=/home/impadmin
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,103] INFO Client
> environment:user.dir=/home/impadmin/Software/kafka_2.8.0-0.8.0-beta1
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,104] INFO Initiating client connection, connectString=
> 192.168.145.144:2181 sessionTimeout=30000
> watcher=org.I0Itec.zkclient.ZkClient@6825c828
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:52,105] DEBUG zookeeper.disableAutoWatchReset is false
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:52,299] DEBUG Awaiting connection to Zookeeper server
> (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:52,299] INFO Opening socket connection to server /
> 192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:52,299] DEBUG Waiting for keeper state SyncConnected
> (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:52,370] INFO Socket connection established to
> hadoop-namenode/192.168.145.144:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:52,372] DEBUG Session establishment request sent on
> hadoop-namenode/192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:52,466] INFO Session establishment complete on server
> hadoop-namenode/192.168.145.144:2181, sessionid = 0x1427e69d4ca0012,
> negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:52,467] DEBUG Received event: WatchedEvent
> state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:52,467] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:52,467] DEBUG Leaving process event
> (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:52,467] DEBUG State is SyncConnected
> (org.I0Itec.zkclient.ZkClient)
> Group Topic Pid Offset
> logSize Lag Owner
> [2013-11-22 11:55:53,002] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
> packet:: clientPath:null serverPath:null finished:false header:: 1,4
> replyHeader:: 1,298,0 request:: '/brokers/topics/test-topic,F response::
>
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d207d2c202276657273696f6e223a31207d,s{122,122,1385100378749,1385100378749,0,1,0,0,54,1,125}
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,472] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
> packet:: clientPath:null serverPath:null finished:false header:: 2,4
> replyHeader:: 2,304,-101 request::
> '/consumers/consumerGroup675437777/offsets/test-topic/0,F response::
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,478] DEBUG Closing ZkClient...
> (org.I0Itec.zkclient.ZkClient)
> [2013-11-22 11:55:53,478] INFO Terminate ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2013-11-22 11:55:53,479] DEBUG Closing ZooKeeper connected to
> 192.168.145.144:2181 (org.I0Itec.zkclient.ZkConnection)
> [2013-11-22 11:55:53,479] DEBUG Closing session: 0x1427e69d4ca0012
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:53,479] DEBUG Closing client for session:
> 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,501] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
> packet:: clientPath:null serverPath:null finished:false header:: 3,-11
> replyHeader:: 3,307,0 request:: null response:: null
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,501] DEBUG An exception was thrown while closing send
> thread for session 0x1427e69d4ca0012 : Unable to read additional data from
> server sessionid 0x1427e69d4ca0012, likely server has closed socket
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,501] DEBUG Disconnecting client for session:
> 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,502] INFO Session: 0x1427e69d4ca0012 closed
> (org.apache.zookeeper.ZooKeeper)
> [2013-11-22 11:55:53,502] INFO EventThread shut down
> (org.apache.zookeeper.ClientCnxn)
> [2013-11-22 11:55:53,502] DEBUG Closing ZkClient...done
> (org.I0Itec.zkclient.ZkClient)
> Exception in thread "main" org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /consumers/consumerGroup675437777/offsets/test-topic/0
> at
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> at kafka.utils.ZkUtils$.readData(ZkUtils.scala:407)
> at
>
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
> at
>
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:90)
> at
>
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> at
>
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
>
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:89)
> at
>
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> at
>
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:153)
> at kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /consumers/consumerGroup675437777/offsets/test-topic/0
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
> at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
> at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
> at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> ... 16 more
>
>
>
> On Thu, Nov 21, 2013 at 9:58 PM, Jun Rao <jun...@gmail.com> wrote:
>
> > When it doesn't consume which offset is in the fetch request? You can
> find
> > out from the request log in the broker.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Nov 21, 2013 at 5:17 AM, Tarang Dawer <tarang.da...@gmail.com
> > >wrote:
> >
> > > Hello
> > >
> > > I am running a kafka 0.8 consumer with the following configuration : -
> > >
> > > fetch.size=1000000000
> > > zookeeper.session.timeout.ms=60000
> > > auto.offset.reset=smallest
> > > zookeeper.sync.time.ms=200
> > > auto.commit.enable=false
> > >
> > > i am doing manual increment of the offsets.
> > >
> > > while doing so, i am facing a problem as when i clear all the kafka &
> > > zookeeper logs, and start the consumer, the consumer sometimes shows
> > > inconsistent behaviour, sometimes it starts consuming, sometimes it
> > > doesn't.
> > > if it doesn't then i kill the consumer and restart it again, only then
> > the
> > > consumption starts.
> > >
> > > i am quite confused with the consumer behaviour. could somebody please
> > help
> > > me out as to how and where am i going wrong ?
> > >
> > > Thanks & Regards
> > > Tarang Dawer
> > >
> >
>
