Hi Cliff, Short answer to your question is it is just the current implementation.
The coordinator stores the offsets as messages in an internal topic and also keeps the latest offset values in in-memory. It answers ConsumerGroupRequest using its cached offset, and upon the consumer group being removed since no member is alive already, it removed it from its in-memory cache and add a "tombstone" to the offset log as well. But the offsets are still persistent as messages in the log, which will only be compacted after a while (this is depend on the log compaction policy). There is a ticket open for improving on this scenario ( https://issues.apache.org/jira/browse/KAFKA-2720) which lets the coordinator to only "purge" dead groups periodically instead of immediately, and that may partially resolve your case. Guozhang On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne <crh...@signal.co> wrote: > Just following up on this concern. Is there a constraint that prevents > ConsumerGroupCommand from reporting offsets on a group if no members are > connected, or is this just the current implementation? > > Thanks, > Cliff > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > I'm running into a few challenges trying to evaluate offsets and lag > > (pending message count) in the new Java KafkaConsumer. The old > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > stored > > in zookeeper after switching from the old consumer. This would be fine, > > but the kafka-consumer-groups.sh command doesn't work if the consumers > are > > shut off. This seems like an unnecessary limitation and is problematic > for > > troubleshooting / monitoring when the application is turned off (or while > > my application is running due to our stopping/starting consumers). > > > > Is there a constraint that I'm not aware of or is this something that > > could be changed? > > > > Thanks, > > Cliff > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > > > > -- > Cliff Rhyne > Software Engineering Lead > e: crh...@signal.co > signal.co > ________________________ > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2015 Signal. All rights reserved. > -- -- Guozhang
