Thanks , cleared my confusion.
On Nov 25, 2013 10:58 PM, "Guozhang Wang" <wangg...@gmail.com> wrote:
> Hi Tarang,
>
> Details about the issue you brought up can be found in KAFKA-1004.
>
> Guozhang
>
>
> On Mon, Nov 25, 2013 at 7:43 AM, Jun Rao <jun...@gmail.com> wrote:
>
> > If you use the latest code from the 0.8 branch, you should be able to
> start
> > the consumer before a topic is created.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, Nov 24, 2013 at 8:14 PM, Tarang Dawer <tarang.da...@gmail.com
> > >wrote:
> >
> > > okay
> > >
> > > is it necessary that a topic should be created first before a consumer
> is
> > > started to listen from that topic ?
> > > or we can start the consumer 1st before the topic has been created ?
> > >
> > >
> > > On Fri, Nov 22, 2013 at 8:53 PM, Jun Rao <jun...@gmail.com> wrote:
> > >
> > > > That error means the consumer has never checkpointed any offset in
> ZK.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, Nov 21, 2013 at 10:31 PM, Tarang Dawer <
> tarang.da...@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Jun
> > > > >
> > > > > I tried the ConsumerOffsetChecker script for checking the offsets
> > > > >
> > > > > but there also , got an exception , with no information regarding
> the
> > > > > offset. Although when i start the consumer for the second time, it
> > > > consumed
> > > > > the data, but still same response from the offset checker script.
> > > > >
> > > > > Please find the debug- level trace of the console output below.
> > > > >
> > > > >
> > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker \--group
> > > > > consumerGroup675437777 \--zkconnect 192.168.145.144:2181 \--topic
> > > > > test-topic;
> > > > > [2013-11-22 11:55:52,098] DEBUG Creating new ZookKeeper instance to
> > > > connect
> > > > > to 192.168.145.144:2181. (org.I0Itec.zkclient.ZkConnection)
> > > > > [2013-11-22 11:55:52,098] INFO Starting ZkClient event thread.
> > > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011
> > 05:47
> > > > GMT
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client environment:host.name
> > > > > =hadoop-namenode
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > environment:java.version=1.6.0_33
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client environment:java.vendor=Sun
> > > > > Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > > environment:java.home=/usr/java/jdk1.6.0_33/jre
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.class.path=:bin/../core/target/scala-2.8.0/*.jar:bin/../perf/target/scala-2.8.0/kafka*.jar:bin/../libs/jopt-simple-3.2.jar:bin/../libs/log4j-1.2.15.jar:bin/../libs/metrics-annotation-2.2.0.jar:bin/../libs/metrics-core-2.2.0.jar:bin/../libs/scala-compiler.jar:bin/../libs/scala-library.jar:bin/../libs/slf4j-api-1.7.2.jar:bin/../libs/slf4j-simple-1.6.4.jar:bin/../libs/snappy-java-1.0.4.1.jar:bin/../libs/zkclient-0.3.jar:bin/../libs/zookeeper-3.3.4.jar:bin/../kafka_2.8.0-0.8.0-beta1.jar
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.library.path=/usr/java/jdk1.6.0_33/jre/lib/amd64/server:/usr/java/jdk1.6.0_33/jre/lib/amd64:/usr/java/jdk1.6.0_33/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> environment:java.io.tmpdir=/tmp
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> environment:java.compiler=<NA>
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client environment:os.name=Linux
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client environment:os.arch=amd64
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client environment:user.name
> =impadmin
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > environment:user.home=/home/impadmin
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,103] INFO Client
> > > > >
> environment:user.dir=/home/impadmin/Software/kafka_2.8.0-0.8.0-beta1
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,104] INFO Initiating client connection,
> > > > connectString=
> > > > > 192.168.145.144:2181 sessionTimeout=30000
> > > > > watcher=org.I0Itec.zkclient.ZkClient@6825c828
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:52,105] DEBUG zookeeper.disableAutoWatchReset is
> > > false
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:52,299] DEBUG Awaiting connection to Zookeeper
> > server
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:52,299] INFO Opening socket connection to server
> /
> > > > > 192.168.145.144:2181 (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:52,299] DEBUG Waiting for keeper state
> > SyncConnected
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:52,370] INFO Socket connection established to
> > > > > hadoop-namenode/192.168.145.144:2181, initiating session
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:52,372] DEBUG Session establishment request sent
> on
> > > > > hadoop-namenode/192.168.145.144:2181
> (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:52,466] INFO Session establishment complete on
> > server
> > > > > hadoop-namenode/192.168.145.144:2181, sessionid =
> 0x1427e69d4ca0012,
> > > > > negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:52,467] DEBUG Received event: WatchedEvent
> > > > > state:SyncConnected type:None path:null
> > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:52,467] INFO zookeeper state changed
> > (SyncConnected)
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:52,467] DEBUG Leaving process event
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:52,467] DEBUG State is SyncConnected
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > Group Topic Pid Offset
> > > > > logSize Lag Owner
> > > > > [2013-11-22 11:55:53,002] DEBUG Reading reply
> > > > sessionid:0x1427e69d4ca0012,
> > > > > packet:: clientPath:null serverPath:null finished:false header::
> 1,4
> > > > > replyHeader:: 1,298,0 request:: '/brokers/topics/test-topic,F
> > > > response::
> > > > >
> > > > >
> > > >
> > >
> >
> #7b2022706172746974696f6e73223a7b202230223a5b2030205d2c202231223a5b2030205d207d2c202276657273696f6e223a31207d,s{122,122,1385100378749,1385100378749,0,1,0,0,54,1,125}
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,472] DEBUG Reading reply
> > > > sessionid:0x1427e69d4ca0012,
> > > > > packet:: clientPath:null serverPath:null finished:false header::
> 2,4
> > > > > replyHeader:: 2,304,-101 request::
> > > > > '/consumers/consumerGroup675437777/offsets/test-topic/0,F
> response::
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,478] DEBUG Closing ZkClient...
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > [2013-11-22 11:55:53,478] INFO Terminate ZkClient event thread.
> > > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > > [2013-11-22 11:55:53,479] DEBUG Closing ZooKeeper connected to
> > > > > 192.168.145.144:2181 (org.I0Itec.zkclient.ZkConnection)
> > > > > [2013-11-22 11:55:53,479] DEBUG Closing session: 0x1427e69d4ca0012
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:53,479] DEBUG Closing client for session:
> > > > > 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,501] DEBUG Reading reply
> > > > sessionid:0x1427e69d4ca0012,
> > > > > packet:: clientPath:null serverPath:null finished:false header::
> > 3,-11
> > > > > replyHeader:: 3,307,0 request:: null response:: null
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,501] DEBUG An exception was thrown while
> closing
> > > > send
> > > > > thread for session 0x1427e69d4ca0012 : Unable to read additional
> data
> > > > from
> > > > > server sessionid 0x1427e69d4ca0012, likely server has closed socket
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,501] DEBUG Disconnecting client for session:
> > > > > 0x1427e69d4ca0012 (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,502] INFO Session: 0x1427e69d4ca0012 closed
> > > > > (org.apache.zookeeper.ZooKeeper)
> > > > > [2013-11-22 11:55:53,502] INFO EventThread shut down
> > > > > (org.apache.zookeeper.ClientCnxn)
> > > > > [2013-11-22 11:55:53,502] DEBUG Closing ZkClient...done
> > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > Exception in thread "main"
> > > > org.I0Itec.zkclient.exception.ZkNoNodeException:
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > KeeperErrorCode =
> > > > > NoNode for /consumers/consumerGroup675437777/offsets/test-topic/0
> > > > > at
> > > > >
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> > > > > at
> > > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> > > > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> > > > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> > > > > at kafka.utils.ZkUtils$.readData(ZkUtils.scala:407)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:90)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:90)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > > > > at scala.collection.immutable.List.foreach(List.scala:45)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:89)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:154)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> > > > > at scala.collection.immutable.List.foreach(List.scala:45)
> > > > > at
> > > > >
> > >
> kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:153)
> > > > > at
> > > > kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
> > > > > Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > KeeperErrorCode = NoNode for
> > > > > /consumers/consumerGroup675437777/offsets/test-topic/0
> > > > > at
> > > > >
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> > > > > at
> > > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
> > > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
> > > > > at
> > org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
> > > > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
> > > > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
> > > > > at
> > > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> > > > > ... 16 more
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Nov 21, 2013 at 9:58 PM, Jun Rao <jun...@gmail.com> wrote:
> > > > >
> > > > > > When it doesn't consume which offset is in the fetch request? You
> > can
> > > > > find
> > > > > > out from the request log in the broker.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 21, 2013 at 5:17 AM, Tarang Dawer <
> > > tarang.da...@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Hello
> > > > > > >
> > > > > > > I am running a kafka 0.8 consumer with the following
> > configuration
> > > :
> > > > -
> > > > > > >
> > > > > > > fetch.size=1000000000
> > > > > > > zookeeper.session.timeout.ms=60000
> > > > > > > auto.offset.reset=smallest
> > > > > > > zookeeper.sync.time.ms=200
> > > > > > > auto.commit.enable=false
> > > > > > >
> > > > > > > i am doing manual increment of the offsets.
> > > > > > >
> > > > > > > while doing so, i am facing a problem as when i clear all the
> > > kafka &
> > > > > > > zookeeper logs, and start the consumer, the consumer sometimes
> > > shows
> > > > > > > inconsistent behaviour, sometimes it starts consuming,
> sometimes
> > > it
> > > > > > > doesn't.
> > > > > > > if it doesn't then i kill the consumer and restart it again,
> only
> > > > then
> > > > > > the
> > > > > > > consumption starts.
> > > > > > >
> > > > > > > i am quite confused with the consumer behaviour. could somebody
> > > > please
> > > > > > help
> > > > > > > me out as to how and where am i going wrong ?
> > > > > > >
> > > > > > > Thanks & Regards
> > > > > > > Tarang Dawer
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
