Brokers: 0.9.0.1 Consumers: 0.8.2.2 In the normal situation my monitoring system runs the consumer groups tool to check consumer offsets.
Example: [ac...@ekk001.atl kafka]$ sudo /opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-consumer-groups.sh --zookeeper ekz003.atl:2181 --describe --group indexers GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER indexers, logs, 0, 4434574869, 4435962019, 1387150, indexers_lsi003.atl-1462602067343-cbd9e429-0 indexers, logs, 1, 4434493999, 4435881150, 1387151, indexers_lsi004.atl-1462602071523-ebdd27b7-0 indexers, logs, 2, 4434579419, 4435966570, 1387151, indexers_lsi005.atl-1462602075179-d77875f3-0 indexers, logs, 3, 4434505990, 4435893141, 1387151, indexers_lsi006.atl-1462602078480-1d3c6104-0 indexers, logs, 4, 4101016475, 4102403639, 1387164, indexers_lsi007.atl-1462602088501-6dad3dcf-0 indexers, logs, 5, 2574701558, 2576088735, 1387177, indexers_lsi009.atl-1462602093014-6463b293-0 All has been running perfect until tonight. I upgraded one of the consumers to 0.9.0.1. Things got really weird in that the one 0.9.0.1 consumer seemed to be it's own consumer group even though the ID is the same. I rolled back the change and now my offsets are not updating in that tool. When i check zookeeper, i confirmed that the consumers are properly updating zookeeper [ac...@ekk001.atl kafka]$ sudo /opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect ekz001.atl:2181,ekz002.atl:2181 --group indexers --output-file ~/offsets [ac...@ekk001.atl kafka]$ cat ~/offsets /consumers/indexers/offsets/logs/0:4435995793 /consumers/indexers/offsets/logs/1:4435914942 /consumers/indexers/offsets/logs/2:4436000590 /consumers/indexers/offsets/logs/3:4435926933 /consumers/indexers/offsets/logs/4:4102437418 /consumers/indexers/offsets/logs/5:2576122730 I already tried stopping all the consumers and deleting the zookeeper path for consumers. Data is flowing properly to the downstream servers but i would like the monitoring to the fixed. Does anyone know how i can fix this offset issue on the consumer_groups tool? Thanks for your attention -- Allen Michael Chan
