Hi, all
We have upgraded Kafka from 0.8.2 to 0.10.2. In Kafka 0.8.2, we store
offset for each consumer-group in Zookeeper. We used Python API to get
offset for each consumer-group in ZK to monitor the lag for each Topic.
After upgraded to Kafka 0.10.2, we plan to store consumer-group offset in
Kafka itself. But we need to using Python API to monitor the consumer
offset. We need some Python API like the kafka-consumer-groups.sh tool to
list all consumer-group & show their offset one by one.
We plan to use confluent-kafka-python/librdkafka as our option. But we can
not find API to list all consumer groups. For the offset for each
consumer-group, we find it can use position method.
Here is our code:
consumer = confluent_kafka.Consumer(conf)
consumer.subscribe(['xxx'])
p = confluent_kafka.TopicPartition("xxx", 1)
print consumer.position([p])
result:
[TopicPartition{topic=elk_app,partition=1,offset=-1001,error=None}]
Seems it can not show the correct offset.
Any thing I did wrong?
In sum, Here is my 2 questions:
1. How to list all consumer-groups using How to list all
consumer-groups/librdkafka API for a topic?
2. How to get current offset for a consumer-group?
Thanks a lot.
--
Yuxiang Mai
Sun Yat-Sen Unitversity
State Key Lab of Optoelectronic
Materials and Technologies
