Hi Tarang,
Details about the issue you brought up can be found in KAFKA-1004.
Guozhang
On Mon, Nov 25, 2013 at 7:43 AM, Jun Rao <jun...@gmail.com> wrote:
> If you use the latest code from the 0.8 branch, you should be able to start
> the consumer before a topic is created.
>
> Thanks,
>
> Jun
>
>
> On Sun, Nov 24, 2013 at 8:14 PM, Tarang Dawer <tarang.da...@gmail.com
> >wrote:
>
> > okay
> >
> > is it necessary that a topic should be created first before a consumer is
> > started to listen from that topic ?
> > or we can start the consumer 1st before the topic has been created ?
> >
> >
> > On Fri, Nov 22, 2013 at 8:53 PM, Jun Rao <jun...@gmail.com> wrote:
> >
> > > That error means the consumer has never checkpointed any offset in ZK.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Nov 21, 2013 at 10:31 PM, Tarang Dawer <tarang.da...@gmail.com
> > > >wrote:
> > >
> > > > Hi Jun
> > > >
> > > > I tried the ConsumerOffsetChecker script for checking the offsets
> > > >
> > > > but there also , got an exception , with no information regarding the
> > > > offset. Although when i start the consumer for the second time, it
> > > consumed
> > > > the data, but still same response from the offset checker script.
> > > >
> > > > Please find the debug- level trace of the console output below.
> > > >
> > > >
> > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker \--group
> > > > consumerGroup675437777 \--zkconnect 192.168.145.144:2181 \--topic
> > > > test-topic;
> > > > [2013-11-22 11:55:52,098] DEBUG Creating new ZookKeeper instance to
> > > connect
> > > > to 192.168.145.144:2181. (org.I0Itec.zkclient.ZkConnection)
> > > > [2013-11-22 11:55:52,098] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011
> 05:47
> > > GMT
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:host.name
> > > > =hadoop-namenode
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> environment:java.version=1.6.0_33
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:java.vendor=Sun
> > > > Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > > environment:java.home=/usr/java/jdk1.6.0_33/jre
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > >
> > > >
> > >
> >
> environment:java.class.path=:bin/../core/target/scala-2.8.0/*.jar:bin/../perf/target/scala-2.8.0/kafka*.jar:bin/../libs/jopt-simple-3.2.jar:bin/../libs/log4j-1.2.15.jar:bin/../libs/metrics-annotation-2.2.0.jar:bin/../libs/metrics-core-2.2.0.jar:bin/../libs/scala-compiler.jar:bin/../libs/scala-library.jar:bin/../libs/slf4j-api-1.7.2.jar:bin/../libs/slf4j-simple-1.6.4.jar:bin/../libs/snappy-java-1.0.4.1.jar:bin/../libs/zkclient-0.3.jar:bin/../libs/zookeeper-3.3.4.jar:bin/../kafka_2.8.0-0.8.0-beta1.jar
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > >
> > > >
> > >
> >
> environment:java.library.path=/usr/java/jdk1.6.0_33/jre/lib/amd64/server:/usr/java/jdk1.6.0_33/jre/lib/amd64:/usr/java/jdk1.6.0_33/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:java.io.tmpdir=/tmp
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:java.compiler=<NA>
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:os.name=Linux
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:os.arch=amd64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client environment:user.name=impadmin
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > environment:user.home=/home/impadmin
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,103] INFO Client
> > > > environment:user.dir=/home/impadmin/Software/kafka_2.8.0-0.8.0-beta1
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,104] INFO Initiating client connection,
> > > connectString=
> > > > 192.168.145.144:2181 sessionTimeout=30000
> > > > watcher=org.I0Itec.zkclient.ZkClient@6825c828
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:52,105] DEBUG zookeeper.disableAutoWatchReset is
> > false
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:52,299] DEBUG Awaiting connection to Zookeeper
> server
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:52,299] INFO Opening socket connection to server /
> > > > 192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:52,299] DEBUG Waiting for keeper state
> SyncConnected
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:52,370] INFO Socket connection established to
> > > > hadoop-namenode/192.168.145.144:2181, initiating session
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:52,372] DEBUG Session establishment request sent on
> > > > hadoop-namenode/192.168.145.144:2181(org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:52,466] INFO Session establishment complete on
> server
> > > > hadoop-namenode/192.168.145.144:2181, sessionid = 0x1427e69d4ca0012,
> > > > negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:52,467] DEBUG Received event: WatchedEvent
> > > > state:SyncConnected type:None path:null
> (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:52,467] INFO zookeeper state changed
> (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:52,467] DEBUG Leaving process event
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:52,467] DEBUG State is SyncConnected
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > Group Topic Pid Offset
> > > > logSize Lag Owner
> > > > [2013-11-22 11:55:53,002] DEBUG Reading reply
> > > sessionid:0x1427e69d4ca0012,
> > > > packet:: clientPath:null serverPath:null finished:false header:: 1,4
> > > > replyHeader:: 1,298,0 request:: '/brokers/topics/test-topic,F
> > > response::
> > > >
> > > >
> > >
> >
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d207d2c202276657273696f6e223a31207d,s{122,122,1385100378749,1385100378749,0,1,0,0,54,1,125}
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,472] DEBUG Reading reply
> > > sessionid:0x1427e69d4ca0012,
> > > > packet:: clientPath:null serverPath:null finished:false header:: 2,4
> > > > replyHeader:: 2,304,-101 request::
> > > > '/consumers/consumerGroup675437777/offsets/test-topic/0,F response::
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,478] DEBUG Closing ZkClient...
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-11-22 11:55:53,478] INFO Terminate ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2013-11-22 11:55:53,479] DEBUG Closing ZooKeeper connected to
> > > > 192.168.145.144:2181 (org.I0Itec.zkclient.ZkConnection)
> > > > [2013-11-22 11:55:53,479] DEBUG Closing session: 0x1427e69d4ca0012
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:53,479] DEBUG Closing client for session:
> > > > 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,501] DEBUG Reading reply
> > > sessionid:0x1427e69d4ca0012,
> > > > packet:: clientPath:null serverPath:null finished:false header::
> 3,-11
> > > > replyHeader:: 3,307,0 request:: null response:: null
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,501] DEBUG An exception was thrown while closing
> > > send
> > > > thread for session 0x1427e69d4ca0012 : Unable to read additional data
> > > from
> > > > server sessionid 0x1427e69d4ca0012, likely server has closed socket
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,501] DEBUG Disconnecting client for session:
> > > > 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,502] INFO Session: 0x1427e69d4ca0012 closed
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2013-11-22 11:55:53,502] INFO EventThread shut down
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2013-11-22 11:55:53,502] DEBUG Closing ZkClient...done
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > Exception in thread "main"
> > > org.I0Itec.zkclient.exception.ZkNoNodeException:
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode =
> > > > NoNode for /consumers/consumerGroup675437777/offsets/test-topic/0
> > > > at
> > > > org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > > > at
> > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > > > at kafka.utils.ZkUtils$.readData(ZkUtils.scala:407)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:90)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> > > > at
> > > >
> > > >
> > >
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > > > at scala.collection.immutable.List.foreach(List.scala:45)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:89)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> > > > at
> > > >
> > > >
> > >
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > > > at scala.collection.immutable.List.foreach(List.scala:45)
> > > > at
> > > >
> > kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:153)
> > > > at
> > > kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
> > > > Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> > > > KeeperErrorCode = NoNode for
> > > > /consumers/consumerGroup675437777/offsets/test-topic/0
> > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> > > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
> > > > at
> org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
> > > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
> > > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
> > > > at
> > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> > > > ... 16 more
> > > >
> > > >
> > > >
> > > > On Thu, Nov 21, 2013 at 9:58 PM, Jun Rao <jun...@gmail.com> wrote:
> > > >
> > > > > When it doesn't consume which offset is in the fetch request? You
> can
> > > > find
> > > > > out from the request log in the broker.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, Nov 21, 2013 at 5:17 AM, Tarang Dawer <
> > tarang.da...@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hello
> > > > > >
> > > > > > I am running a kafka 0.8 consumer with the following
> configuration
> > :
> > > -
> > > > > >
> > > > > > fetch.size=1000000000
> > > > > > zookeeper.session.timeout.ms=60000
> > > > > > auto.offset.reset=smallest
> > > > > > zookeeper.sync.time.ms=200
> > > > > > auto.commit.enable=false
> > > > > >
> > > > > > i am doing manual increment of the offsets.
> > > > > >
> > > > > > while doing so, i am facing a problem as when i clear all the
> > kafka &
> > > > > > zookeeper logs, and start the consumer, the consumer sometimes
> > shows
> > > > > > inconsistent behaviour, sometimes it starts consuming, sometimes
> > it
> > > > > > doesn't.
> > > > > > if it doesn't then i kill the consumer and restart it again, only
> > > then
> > > > > the
> > > > > > consumption starts.
> > > > > >
> > > > > > i am quite confused with the consumer behaviour. could somebody
> > > please
> > > > > help
> > > > > > me out as to how and where am i going wrong ?
> > > > > >
> > > > > > Thanks & Regards
> > > > > > Tarang Dawer
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
-- Guozhang
