Good point. I think it makes sense to remove the corresponding metric if the consumer does not own a partition anymore. I have updated the KIP.
On Wed, Dec 21, 2016 at 10:24 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > LGTM. However, can you comment on the effect of releasing ownership of > partitions after a rebalance? For e.g., should it reset itself to (say) -1? > or removed? This really applies to any per-partition metrics that we intend > to maintain in the consumer. > > On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin <becket....@gmail.com> wrote: > > > Hey Michael, > > > > Thanks for the comments. Exposing the lag on the client side may serve > some > > cases a little different from monitoring. For example, one of the use > case > > we have was that an application has some high priority and low priority > > topics to consume. They want to switch between consuming from high > priority > > and low priority topics alternately based on the lag on the partition. > i.e. > > if the consume has already caught up with the log end offset of a high > > priority topic, the application will switch to consume from the low > > priority topics. Otherwise it will continue consuming from the high > > priority topics. We have seen a few other similar use cases that require > a > > programmatic access to the lag. Although people can always use > > offsetsForTimes() to get the LEO, but it is more expensive call involving > > an RPC and is a blocking call. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > Should state I have no objections adding this client side, just more a > > > question to why we don't look and propose to add this broker side also. > > > > > > Sent using OWA for iPhone > > > ________________________________________ > > > From: Michael Pearce <michael.pea...@ig.com> > > > Sent: Monday, November 14, 2016 4:58:45 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to > > > KafkaConsumer > > > > > > Why do we not look to expose the lag broker side centrally? > > > > > > Eg like burrow. > > > > > > From an operations point it's a lot easier to monitor lag centrally > than > > > per application. Also then you'd be able to see lag of consumers not > > alive > > > or stalled. > > > > > > The information if the consumer uses Kafka based or zookeeper offsets > is > > > available to the broker. > > > ________________________________________ > > > From: Becket Qin <becket....@gmail.com> > > > Sent: Sunday, November 13, 2016 4:13:01 AM > > > To: dev@kafka.apache.org > > > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to > > KafkaConsumer > > > > > > Hi, > > > > > > We created KIP-92 to propose adding per partition lag metrics to > > > KafkaConsumer. > > > > > > The KIP wiki link is the following: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer > > > > > > Comments are welcome. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > The information contained in this email is strictly confidential and > for > > > the use of the addressee only, unless otherwise indicated. If you are > not > > > the intended recipient, please do not read, copy, use or disclose to > > others > > > this message or any attachment. Please also notify the sender by > replying > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > email > > > and any copies of it. Opinions, conclusion (etc) that do not relate to > > the > > > official business of this company shall be understood as neither given > > nor > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > registered in England and Wales, company number 04008957) and IG Index > > > Limited (a company registered in England and Wales, company number > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > IG > > > Index Limited (register number 114059) are authorised and regulated by > > the > > > Financial Conduct Authority. > > > > > >
