Thanks for the added info. For the mean time we'll rely on the older ConsumerOffsetChecker for our monitoring tools.
Thanks, Cliff On Fri, Jan 29, 2016 at 10:56 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Tao, > > You are right, ConsumerOffsetChecker can still get offsets from the offset > manager in Kafka. > > Guozhang > > On Thu, Jan 28, 2016 at 9:36 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > it first issues an offsetrequest to broker and check if offset is stored > in > > broker if not it will queries zk > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171 > > > > On Fri, 29 Jan 2016 at 13:11 Guozhang Wang <wangg...@gmail.com> wrote: > > > > > Tao, > > > > > > Hmm that is a bit wired since ConsumerOffsetChecker itself does not > talk > > to > > > brokers at all, but only through ZK. > > > > > > Guozhang > > > > > > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao <xiaotao...@gmail.com> > wrote: > > > > > > > Guozhang, > > > > > > > > The old ConsumerOffsetChecker works for new consumer too with offset > > > stored > > > > in Kafka. I tested it with mirror maker with new consumer enabled. it > > is > > > > able to show offset during mirror maker running and after its > shutdown. > > > > > > > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > > > > > Once the offset is written to the log it is persistent and hence > > should > > > > > survive broker failures. And its retention policy is configurable. > > > > > > > > > > It may be a bit misleading in saying "in-memory cache" in my > previous > > > > > email: the brokers just keep the in-memory map of [group, > partition] > > -> > > > > > latest_offset, while the offset commit history is kept in the log. > > When > > > > we > > > > > delete the group, we remove the corresponding entry from memory map > > and > > > > put > > > > > a tombstone into log as well so that the old offsets will be > > compacted > > > > > eventually according to compaction policy. > > > > > > > > > > The old ConsumerOffsetChecker only works for old consumer that > stores > > > > > offset in ZK. > > > > > > > > > > Guozhang > > > > > > > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne <crh...@signal.co> > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > That looks like it might help but feels like there might be some > > > gaps. > > > > > > Would it be able to survive restarts of the kafka broker? How > long > > > > would > > > > > > it stay in the cache (and is that configurable)? If it expires > > from > > > > the > > > > > > cache, what's the cache-miss operation look like? (yes, a lot of > > > this > > > > > > depends on the data still being in the logs to recover) > > > > > > > > > > > > In the mean time, can I rely on the deprecated > > ConsumerOffsetChecker > > > > > (which > > > > > > looks at zookeeper) even though I'm using the new KafkaConsumer? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang < > wangg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Cliff, > > > > > > > > > > > > > > Short answer to your question is it is just the current > > > > implementation. > > > > > > > > > > > > > > The coordinator stores the offsets as messages in an internal > > topic > > > > and > > > > > > > also keeps the latest offset values in in-memory. It answers > > > > > > > ConsumerGroupRequest using its cached offset, and upon the > > consumer > > > > > group > > > > > > > being removed since no member is alive already, it removed it > > from > > > > its > > > > > > > in-memory cache and add a "tombstone" to the offset log as > well. > > > But > > > > > the > > > > > > > offsets are still persistent as messages in the log, which will > > > only > > > > be > > > > > > > compacted after a while (this is depend on the log compaction > > > > policy). > > > > > > > > > > > > > > There is a ticket open for improving on this scenario ( > > > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets > the > > > > > > > coordinator to only "purge" dead groups periodically instead of > > > > > > > immediately, and that may partially resolve your case. > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne < > crh...@signal.co> > > > > > wrote: > > > > > > > > > > > > > > > Just following up on this concern. Is there a constraint > that > > > > > prevents > > > > > > > > ConsumerGroupCommand from reporting offsets on a group if no > > > > members > > > > > > are > > > > > > > > connected, or is this just the current implementation? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Cliff > > > > > > > > > > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne < > crh...@signal.co > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I'm running into a few challenges trying to evaluate > offsets > > > and > > > > > lag > > > > > > > > > (pending message count) in the new Java KafkaConsumer. The > > old > > > > > > > > > ConsumerOffsetChecker doesn't work anymore since the > offsets > > > > aren't > > > > > > > > stored > > > > > > > > > in zookeeper after switching from the old consumer. This > > would > > > > be > > > > > > > fine, > > > > > > > > > but the kafka-consumer-groups.sh command doesn't work if > the > > > > > > consumers > > > > > > > > are > > > > > > > > > shut off. This seems like an unnecessary limitation and is > > > > > > problematic > > > > > > > > for > > > > > > > > > troubleshooting / monitoring when the application is turned > > off > > > > (or > > > > > > > while > > > > > > > > > my application is running due to our stopping/starting > > > > consumers). > > > > > > > > > > > > > > > > > > Is there a constraint that I'm not aware of or is this > > > something > > > > > that > > > > > > > > > could be changed? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Cliff > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Cliff Rhyne > > > > > > > > > Software Engineering Lead > > > > > > > > > e: crh...@signal.co > > > > > > > > > signal.co > > > > > > > > > ________________________ > > > > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > > > > > > > This e-mail and any files transmitted with it are for the > > sole > > > > use > > > > > of > > > > > > > the > > > > > > > > > intended recipient(s) and may contain confidential and > > > privileged > > > > > > > > > information. Any unauthorized use of this email is strictly > > > > > > prohibited. > > > > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Cliff Rhyne > > > > > > > > Software Engineering Lead > > > > > > > > e: crh...@signal.co > > > > > > > > signal.co > > > > > > > > ________________________ > > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > > > > > This e-mail and any files transmitted with it are for the > sole > > > use > > > > of > > > > > > the > > > > > > > > intended recipient(s) and may contain confidential and > > privileged > > > > > > > > information. Any unauthorized use of this email is strictly > > > > > prohibited. > > > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Cliff Rhyne > > > > > > Software Engineering Lead > > > > > > e: crh...@signal.co > > > > > > signal.co > > > > > > ________________________ > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole > use > > of > > > > the > > > > > > intended recipient(s) and may contain confidential and privileged > > > > > > information. Any unauthorized use of this email is strictly > > > prohibited. > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
