Guozhang, The old ConsumerOffsetChecker works for new consumer too with offset stored in Kafka. I tested it with mirror maker with new consumer enabled. it is able to show offset during mirror maker running and after its shutdown.
On Fri, 29 Jan 2016 at 06:34 Guozhang Wang <wangg...@gmail.com> wrote: > Once the offset is written to the log it is persistent and hence should > survive broker failures. And its retention policy is configurable. > > It may be a bit misleading in saying "in-memory cache" in my previous > email: the brokers just keep the in-memory map of [group, partition] -> > latest_offset, while the offset commit history is kept in the log. When we > delete the group, we remove the corresponding entry from memory map and put > a tombstone into log as well so that the old offsets will be compacted > eventually according to compaction policy. > > The old ConsumerOffsetChecker only works for old consumer that stores > offset in ZK. > > Guozhang > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > Hi Guozhang, > > > > That looks like it might help but feels like there might be some gaps. > > Would it be able to survive restarts of the kafka broker? How long would > > it stay in the cache (and is that configurable)? If it expires from the > > cache, what's the cache-miss operation look like? (yes, a lot of this > > depends on the data still being in the logs to recover) > > > > In the mean time, can I rely on the deprecated ConsumerOffsetChecker > (which > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > Thanks, > > Cliff > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hi Cliff, > > > > > > Short answer to your question is it is just the current implementation. > > > > > > The coordinator stores the offsets as messages in an internal topic and > > > also keeps the latest offset values in in-memory. It answers > > > ConsumerGroupRequest using its cached offset, and upon the consumer > group > > > being removed since no member is alive already, it removed it from its > > > in-memory cache and add a "tombstone" to the offset log as well. But > the > > > offsets are still persistent as messages in the log, which will only be > > > compacted after a while (this is depend on the log compaction policy). > > > > > > There is a ticket open for improving on this scenario ( > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the > > > coordinator to only "purge" dead groups periodically instead of > > > immediately, and that may partially resolve your case. > > > > > > Guozhang > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne <crh...@signal.co> > wrote: > > > > > > > Just following up on this concern. Is there a constraint that > prevents > > > > ConsumerGroupCommand from reporting offsets on a group if no members > > are > > > > connected, or is this just the current implementation? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne <crh...@signal.co> > wrote: > > > > > > > > > I'm running into a few challenges trying to evaluate offsets and > lag > > > > > (pending message count) in the new Java KafkaConsumer. The old > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets aren't > > > > stored > > > > > in zookeeper after switching from the old consumer. This would be > > > fine, > > > > > but the kafka-consumer-groups.sh command doesn't work if the > > consumers > > > > are > > > > > shut off. This seems like an unnecessary limitation and is > > problematic > > > > for > > > > > troubleshooting / monitoring when the application is turned off (or > > > while > > > > > my application is running due to our stopping/starting consumers). > > > > > > > > > > Is there a constraint that I'm not aware of or is this something > that > > > > > could be changed? > > > > > > > > > > Thanks, > > > > > Cliff > > > > > > > > > > -- > > > > > Cliff Rhyne > > > > > Software Engineering Lead > > > > > e: crh...@signal.co > > > > > signal.co > > > > > ________________________ > > > > > > > > > > Cut Through the Noise > > > > > > > > > > This e-mail and any files transmitted with it are for the sole use > of > > > the > > > > > intended recipient(s) and may contain confidential and privileged > > > > > information. Any unauthorized use of this email is strictly > > prohibited. > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > ________________________ > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > > > > -- > -- Guozhang >
