If you can validate these partitions have data (i.e. there are some offsets committed to Kafka), then you may have to turn of debug level logging in configs/tools-log4j.properties which will allow console consumer to print debug level logs and see if there is anything suspicious.
Guozhang On Thu, Dec 3, 2015 at 10:55 AM, Marina <ppi...@yahoo.com.invalid> wrote: > Hi, Guozhang, > Yes, I can see this topic and partitions in ZK: > > ls /brokers/topics/__consumer_offsets > [partitions] > ls /brokers/topics/__consumer_offsets/partitions > [44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2, > 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43] > > > example of one partition state: > ls /brokers/topics/__consumer_offsets/partitions/44 > [state] > ls /brokers/topics/__consumer_offsets/partitions/44/state > [] > > > anything else you think I could check? > > thanks! > Marina > > > > ----- Original Message ----- > From: Guozhang Wang <wangg...@gmail.com> > To: "users@kafka.apache.org" <users@kafka.apache.org>; Marina < > ppi...@yahoo.com> > Sent: Thursday, December 3, 2015 12:05 PM > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic? > > Marina, > > To check if the topic does exist in Kafka (i.e. offsets are stored in Kafka > instead of in ZK) you can check this path in ZK: > > /brokers/topics/__consumer_offsets > > By default this topic should have 50 partitions. > > Guozhang > > > > On Thu, Dec 3, 2015 at 6:22 AM, Marina <ppi...@yahoo.com.invalid> wrote: > > > Hi, Jason, > > > > I tried the same command both with specifying a formatter and without - > > same result: > > > > => /opt/kafka/bin/kafka-console-consumer.sh --formatter > > kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config > > /tmp/consumer.properties --topic __consumer_offsets --zookeeper > > localhost:2181 --from-beginning > > > > ^CConsumed 0 messages > > > > > > When you are saying to make sure Consumers are using Kafka instead of ZK > > storage for offset... - what is the best way to do that? I thought > checking > > the content via ZK shell - at this path: > > /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number> > > > > would be a definitive confirmation (there are no values stored there). Is > > it true? > > > > I feel it is some small stupid mistake I'm making, since it seems to work > > fine for others - drives me crazy :) > > > > thanks!! > > Marina > > > > > > > > ----- Original Message ----- > > From: Jason Gustafson <ja...@confluent.io> > > To: users@kafka.apache.org > > Sent: Wednesday, December 2, 2015 2:33 PM > > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic? > > > > Hey Marina, > > > > My mistake, I see you're using 0.8.2.1. Are you also providing the > > formatter argument when using console-consumer.sh? Perhaps something like > > this: > > > > bin/kafka-console-consumer.sh --formatter > > kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper > > localhost:2181 --topic __consumer_offsets --from-beginning > > > > You may also want to confirm that your consumers are using Kafka instead > of > > Zookeeper for offset storage. If you still don't see anything, we can > > always look into the partition data directly... > > > > -Jason > > > > > > On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > Looks like you need to use a different MessageFormatter class, since it > > > was renamed in 0.9. Instead use something like > > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter". > > > > > > -Jason > > > > > > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan < > > > d.muralidha...@yottaa.com> wrote: > > > > > >> I have this same problem . Can someone help ? > > >> > > >> --Dhyan > > >> > > >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <ppi...@yahoo.com.invalid> > > wrote: > > >> > > >> > Hello, > > >> > > > >> > I'm trying to find out which offsets my current High-Level consumers > > are > > >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set > in > > >> the > > >> > server.properties of Kafka - which, I think, means that offsets are > > >> stored > > >> > in Kafka. (I also verified that no offsets are stored in Zookeeper > by > > >> > checking this path in the Zk shell: > > >> > > > >> > > > **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>** > > >> > ) > > >> > > > >> > I tried to listen to the **__consumer_offsets** topic to see which > > >> > consumer saves what value of offsets, but it did not work... > > >> > > > >> > I tried the following: > > >> > > > >> > created a config file for console consumer as following: > > >> > > > >> > > > >> > => more kafka_offset_consumer.config > > >> > > > >> > exclude.internal.topics=false` > > >> > > > >> > and tried to versions of the console consumer scripts: > > >> > > > >> > #1: > > >> > bin/kafka-console-consumer.sh --consumer.config > > >> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper > > >> > localhost:2181 > > >> > > > >> > #2 > > >> > ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets > > >> > --partition 0 --broker-list localhost:9092 --formatter > > >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter" > > --consumer.config > > >> > kafka_offset_consumer.config > > >> > > > >> > > > >> > Neither worked - it just sits there but does not print anything, > even > > >> > though the consumers are actively consuming/saving offsets. > > >> > > > >> > Am I missing some other configuration/properties ? > > >> > > > >> > thanks! > > >> > > > >> > Marina > > >> > > > >> > I have also posted this question on StackOverflow: > > >> > > > >> > > > >> > > > >> > > > http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic > > >> > > > >> > > > > > > > > > > > > -- > -- Guozhang > -- -- Guozhang
