Hi Jun
I tried the ConsumerOffsetChecker script for checking the offsets
but there also , got an exception , with no information regarding the
offset. Although when i start the consumer for the second time, it consumed
the data, but still same response from the offset checker script.
Please find the debug- level trace of the console output below.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker \--group
consumerGroup675437777 \--zkconnect 192.168.145.144:2181 \--topic
test-topic;
[2013-11-22 11:55:52,098] DEBUG Creating new ZookKeeper instance to connect
to 192.168.145.144:2181. (org.I0Itec.zkclient.ZkConnection)
[2013-11-22 11:55:52,098] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-11-22 11:55:52,103] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:host.name=hadoop-namenode
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:java.version=1.6.0_33
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:java.vendor=Sun
Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client
environment:java.home=/usr/java/jdk1.6.0_33/jre
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client
environment:java.class.path=:bin/../core/target/scala-2.8.0/*.jar:bin/../perf/target/scala-2.8.0/kafka*.jar:bin/../libs/jopt-simple-3.2.jar:bin/../libs/log4j-1.2.15.jar:bin/../libs/metrics-annotation-2.2.0.jar:bin/../libs/metrics-core-2.2.0.jar:bin/../libs/scala-compiler.jar:bin/../libs/scala-library.jar:bin/../libs/slf4j-api-1.7.2.jar:bin/../libs/slf4j-simple-1.6.4.jar:bin/../libs/snappy-java-1.0.4.1.jar:bin/../libs/zkclient-0.3.jar:bin/../libs/zookeeper-3.3.4.jar:bin/../kafka_2.8.0-0.8.0-beta1.jar
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client
environment:java.library.path=/usr/java/jdk1.6.0_33/jre/lib/amd64/server:/usr/java/jdk1.6.0_33/jre/lib/amd64:/usr/java/jdk1.6.0_33/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client
environment:os.version=2.6.32-279.el6.x86_64
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:user.name=impadmin
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client environment:user.home=/home/impadmin
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,103] INFO Client
environment:user.dir=/home/impadmin/Software/kafka_2.8.0-0.8.0-beta1
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,104] INFO Initiating client connection, connectString=
192.168.145.144:2181 sessionTimeout=30000
watcher=org.I0Itec.zkclient.ZkClient@6825c828(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:52,105] DEBUG zookeeper.disableAutoWatchReset is false
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:52,299] DEBUG Awaiting connection to Zookeeper server
(org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:52,299] INFO Opening socket connection to server /
192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:52,299] DEBUG Waiting for keeper state SyncConnected
(org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:52,370] INFO Socket connection established to
hadoop-namenode/192.168.145.144:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:52,372] DEBUG Session establishment request sent on
hadoop-namenode/192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:52,466] INFO Session establishment complete on server
hadoop-namenode/192.168.145.144:2181, sessionid = 0x1427e69d4ca0012,
negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:52,467] DEBUG Received event: WatchedEvent
state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:52,467] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:52,467] DEBUG Leaving process event
(org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:52,467] DEBUG State is SyncConnected
(org.I0Itec.zkclient.ZkClient)
Group Topic Pid Offset
logSize Lag Owner
[2013-11-22 11:55:53,002] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
packet:: clientPath:null serverPath:null finished:false header:: 1,4
replyHeader:: 1,298,0 request:: '/brokers/topics/test-topic,F response::
#7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d207d2c202276657273696f6e223a31207d,s{122,122,1385100378749,1385100378749,0,1,0,0,54,1,125}
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,472] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
packet:: clientPath:null serverPath:null finished:false header:: 2,4
replyHeader:: 2,304,-101 request::
'/consumers/consumerGroup675437777/offsets/test-topic/0,F response::
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,478] DEBUG Closing ZkClient...
(org.I0Itec.zkclient.ZkClient)
[2013-11-22 11:55:53,478] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-11-22 11:55:53,479] DEBUG Closing ZooKeeper connected to
192.168.145.144:2181 (org.I0Itec.zkclient.ZkConnection)
[2013-11-22 11:55:53,479] DEBUG Closing session: 0x1427e69d4ca0012
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:53,479] DEBUG Closing client for session:
0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,501] DEBUG Reading reply sessionid:0x1427e69d4ca0012,
packet:: clientPath:null serverPath:null finished:false header:: 3,-11
replyHeader:: 3,307,0 request:: null response:: null
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,501] DEBUG An exception was thrown while closing send
thread for session 0x1427e69d4ca0012 : Unable to read additional data from
server sessionid 0x1427e69d4ca0012, likely server has closed socket
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,501] DEBUG Disconnecting client for session:
0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,502] INFO Session: 0x1427e69d4ca0012 closed
(org.apache.zookeeper.ZooKeeper)
[2013-11-22 11:55:53,502] INFO EventThread shut down
(org.apache.zookeeper.ClientCnxn)
[2013-11-22 11:55:53,502] DEBUG Closing ZkClient...done
(org.I0Itec.zkclient.ZkClient)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /consumers/consumerGroup675437777/offsets/test-topic/0
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
at kafka.utils.ZkUtils$.readData(ZkUtils.scala:407)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:90)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:89)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at
kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:153)
at kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/consumers/consumerGroup675437777/offsets/test-topic/0
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 16 more
On Thu, Nov 21, 2013 at 9:58 PM, Jun Rao <jun...@gmail.com> wrote:
> When it doesn't consume which offset is in the fetch request? You can find
> out from the request log in the broker.
>
> Thanks,
>
> Jun
>
>
> On Thu, Nov 21, 2013 at 5:17 AM, Tarang Dawer <tarang.da...@gmail.com
> >wrote:
>
> > Hello
> >
> > I am running a kafka 0.8 consumer with the following configuration : -
> >
> > fetch.size=1000000000
> > zookeeper.session.timeout.ms=60000
> > auto.offset.reset=smallest
> > zookeeper.sync.time.ms=200
> > auto.commit.enable=false
> >
> > i am doing manual increment of the offsets.
> >
> > while doing so, i am facing a problem as when i clear all the kafka &
> > zookeeper logs, and start the consumer, the consumer sometimes shows
> > inconsistent behaviour, sometimes it starts consuming, sometimes it
> > doesn't.
> > if it doesn't then i kill the consumer and restart it again, only then
> the
> > consumption starts.
> >
> > i am quite confused with the consumer behaviour. could somebody please
> help
> > me out as to how and where am i going wrong ?
> >
> > Thanks & Regards
> > Tarang Dawer
> >
>
