*Use Case: Disaster Recovery & Re-indexing SOLR*
I'm using Kafka to hold messages from a service that prepares "documents"
for SOLR.
A second micro service (a consumer) requests these messages, does any final
processing, and fires them into SOLR.
The whole thing is (in part) designed to be used for disaster recovery -
allowing the rebuild of the SOLR index in the shortest possible time.
To do this (and to be able to use it for re-indexing SOLR while testing
relevancy) I need to be able to "play all messages from the beginning" at
will.
I find I can use the zkCli.sh tool to delete the Consumer Group Name like
this:
rmr /kafka/consumers/myGroupName
After which my microservice will get all the messages again when it runs.
I was trying to find a way to do this programmatically without actually
using the "low level" consumer api since the high level one is so simple
and my code already works. So I started playing with Zookeeper api for
duplicating "rmr /kafka/consumers/myGroupName"
*The Question: What does that offset actually represent?*
It was at this point that I discovered the offset must represent something
other than what I thought it would. Things obviously work, but I'm
wondering what - exactly do the offsets represent?
To clarify - if I run this command on a zookeeper node, after the
microservice has run:
get /kafka/consumers/myGroupName/offsets/myTopicName/0
I get the following:
30024
cZxid = 0x3600000355
ctime = Fri Feb 12 07:27:50 MST 2016
mZxid = 0x3600000357
mtime = Fri Feb 12 07:29:50 MST 2016
pZxid = 0x3600000355
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
Now - I have exactly 3500 messages in this Kafka topic. I verify that by
running this command:
bin/kafka-console-consumer.sh --zookeeper 192.168.56.5:2181/kafka
--topic myTopicName --from-beginning
When I hit Ctrl-C, it tells me it consumed 3500 messages.
So - what does that 30024 actually represent? If I reset that number to 1
or 0 and re-run my consumer microservice, I get all the messages again -
and the number again goes to 30024. However, I'm not comfortable to trust
that because my assumption that the number represents a simple count of
messages that have been sent to this consumer is obviously wrong.
(I reset the number like this -- to 1 -- and assume there's an API command
that will do it too.)
set /kafka/consumers/myGroupName/offsets/myTopicName/0 1
Can someone help me clarify or point me at a doc that explains what is
getting counted here? You can shoot me if you like for attempting the
hack-ish solution of re-setting the offset through the Zookeeper API, but I
would still like to understand what, exactly, is represented by that number
30024.
I need to hand off to IT for the Disaster Recovery portion and saying
"trust me, it just works" isn't going to fly very far...
Thanks.
