Thank you, Lance - this is very useful info!I did figure out what was wrong in
my case - the offsets were not stored in KAfka legitimately, they were stored
in Zookeeper, I was using a wrong command to inspect ZK content - doing 'ls
<path>' instead of 'get <path>'.
Once I used 'get' - I could see correct offsets for all my consumers in ZK. We
will switch to using Kafka storage very soon!
Thanks!Marina
From: Lance Laursen <llaur...@rubiconproject.com>
To: users@kafka.apache.org
Cc: Marina <ppi...@yahoo.com>
Sent: Thursday, December 3, 2015 4:35 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Hi Marina,
You can hop onto your brokers and dump your __consumer_offsets logs
manually in order to see if anything is in them. Hop on each of your
brokers and run the following command:
for f in $(find /path/to/kafka-logs/__consumer_offsets-* -name "*\.log");
do /usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments
--print-data-log --files $f ; done
Remember to replace /path/to/kafka-logs with your configured log.dirs
location, and /usr/local/kafka/ with wherever you have installed kafka.
If that command doesn't generate any output (Starting offset: 0 followed by
nothing) on any of your brokers, then your consumers are for some reason
not committing offsets to Kafka.
Keep in mind that commits to __consumer_offsets by the consumer API use
groupID, Topic, and PartitionID to key off of in order to determine which
partition of __consumer_offsets to save the commit message to. This means
that if you're not running many consumers/topics, you could have partitions
with nothing in them and others with lots. Keep this in mind when using
-simple-consumer scripts as the simple consumer only pulls from one
partition at a time.
On Thu, Dec 3, 2015 at 11:10 AM, Guozhang Wang <wangg...@gmail.com> wrote:
> If you can validate these partitions have data (i.e. there are some offsets
> committed to Kafka), then you may have to turn of debug level logging in
> configs/tools-log4j.properties which will allow console consumer to print
> debug level logs and see if there is anything suspicious.
>
> Guozhang
>
> On Thu, Dec 3, 2015 at 10:55 AM, Marina <ppi...@yahoo.com.invalid> wrote:
>
> > Hi, Guozhang,
> > Yes, I can see this topic and partitions in ZK:
> >
> > ls /brokers/topics/__consumer_offsets
> > [partitions]
> > ls /brokers/topics/__consumer_offsets/partitions
> > [44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2,
> > 3, 4, 5, 6, 7, 8, 9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
> > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]
> >
> >
> > example of one partition state:
> > ls /brokers/topics/__consumer_offsets/partitions/44
> > [state]
> > ls /brokers/topics/__consumer_offsets/partitions/44/state
> > []
> >
> >
> > anything else you think I could check?
> >
> > thanks!
> > Marina
> >
> >
> >
> > ----- Original Message -----
> > From: Guozhang Wang <wangg...@gmail.com>
> > To: "users@kafka.apache.org" <users@kafka.apache.org>; Marina <
> > ppi...@yahoo.com>
> > Sent: Thursday, December 3, 2015 12:05 PM
> > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> >
> > Marina,
> >
> > To check if the topic does exist in Kafka (i.e. offsets are stored in
> Kafka
> > instead of in ZK) you can check this path in ZK:
> >
> > /brokers/topics/__consumer_offsets
> >
> > By default this topic should have 50 partitions.
> >
> > Guozhang
> >
> >
> >
> > On Thu, Dec 3, 2015 at 6:22 AM, Marina <ppi...@yahoo.com.invalid> wrote:
> >
> > > Hi, Jason,
> > >
> > > I tried the same command both with specifying a formatter and without -
> > > same result:
> > >
> > > => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> > > /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> > > localhost:2181 --from-beginning
> > >
> > > ^CConsumed 0 messages
> > >
> > >
> > > When you are saying to make sure Consumers are using Kafka instead of
> ZK
> > > storage for offset... - what is the best way to do that? I thought
> > checking
> > > the content via ZK shell - at this path:
> > >
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
> > >
> > > would be a definitive confirmation (there are no values stored there).
> Is
> > > it true?
> > >
> > > I feel it is some small stupid mistake I'm making, since it seems to
> work
> > > fine for others - drives me crazy :)
> > >
> > > thanks!!
> > > Marina
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Jason Gustafson <ja...@confluent.io>
> > > To: users@kafka.apache.org
> > > Sent: Wednesday, December 2, 2015 2:33 PM
> > > Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
> > >
> > > Hey Marina,
> > >
> > > My mistake, I see you're using 0.8.2.1. Are you also providing the
> > > formatter argument when using console-consumer.sh? Perhaps something
> like
> > > this:
> > >
> > > bin/kafka-console-consumer.sh --formatter
> > > kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> > > localhost:2181 --topic __consumer_offsets --from-beginning
> > >
> > > You may also want to confirm that your consumers are using Kafka
> instead
> > of
> > > Zookeeper for offset storage. If you still don't see anything, we can
> > > always look into the partition data directly...
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Looks like you need to use a different MessageFormatter class, since
> it
> > > > was renamed in 0.9. Instead use something like
> > > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > > > d.muralidha...@yottaa.com> wrote:
> > > >
> > > >> I have this same problem . Can someone help ?
> > > >>
> > > >> --Dhyan
> > > >>
> > > >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <ppi...@yahoo.com.invalid>
> > > wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I'm trying to find out which offsets my current High-Level
> consumers
> > > are
> > > >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set
> > in
> > > >> the
> > > >> > server.properties of Kafka - which, I think, means that offsets
> are
> > > >> stored
> > > >> > in Kafka. (I also verified that no offsets are stored in Zookeeper
> > by
> > > >> > checking this path in the Zk shell:
> > > >> >
> > > >>
> > >
> >
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> > > >> > )
> > > >> >
> > > >> > I tried to listen to the **__consumer_offsets** topic to see which
> > > >> > consumer saves what value of offsets, but it did not work...
> > > >> >
> > > >> > I tried the following:
> > > >> >
> > > >> > created a config file for console consumer as following:
> > > >> >
> > > >> >
> > > >> > => more kafka_offset_consumer.config
> > > >> >
> > > >> > exclude.internal.topics=false`
> > > >> >
> > > >> > and tried to versions of the console consumer scripts:
> > > >> >
> > > >> > #1:
> > > >> > bin/kafka-console-consumer.sh --consumer.config
> > > >> > kafka_offset_consumer.config --topic __consumer_offsets
> --zookeeper
> > > >> > localhost:2181
> > > >> >
> > > >> > #2
> > > >> > ./bin/kafka-simple-consumer-shell.sh --topic
> __consumer_offsets
> > > >> > --partition 0 --broker-list localhost:9092 --formatter
> > > >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> > > --consumer.config
> > > >> > kafka_offset_consumer.config
> > > >> >
> > > >> >
> > > >> > Neither worked - it just sits there but does not print anything,
> > even
> > > >> > though the consumers are actively consuming/saving offsets.
> > > >> >
> > > >> > Am I missing some other configuration/properties ?
> > > >> >
> > > >> > thanks!
> > > >> >
> > > >> > Marina
> > > >> >
> > > >> > I have also posted this question on StackOverflow:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>
