LGTM. However, can you comment on the effect of releasing ownership of partitions after a rebalance? For e.g., should it reset itself to (say) -1? or removed? This really applies to any per-partition metrics that we intend to maintain in the consumer.
On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin <becket....@gmail.com> wrote: > Hey Michael, > > Thanks for the comments. Exposing the lag on the client side may serve some > cases a little different from monitoring. For example, one of the use case > we have was that an application has some high priority and low priority > topics to consume. They want to switch between consuming from high priority > and low priority topics alternately based on the lag on the partition. i.e. > if the consume has already caught up with the log end offset of a high > priority topic, the application will switch to consume from the low > priority topics. Otherwise it will continue consuming from the high > priority topics. We have seen a few other similar use cases that require a > programmatic access to the lag. Although people can always use > offsetsForTimes() to get the LEO, but it is more expensive call involving > an RPC and is a blocking call. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce <michael.pea...@ig.com> > wrote: > > > Should state I have no objections adding this client side, just more a > > question to why we don't look and propose to add this broker side also. > > > > Sent using OWA for iPhone > > ________________________________________ > > From: Michael Pearce <michael.pea...@ig.com> > > Sent: Monday, November 14, 2016 4:58:45 PM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to > > KafkaConsumer > > > > Why do we not look to expose the lag broker side centrally? > > > > Eg like burrow. > > > > From an operations point it's a lot easier to monitor lag centrally than > > per application. Also then you'd be able to see lag of consumers not > alive > > or stalled. > > > > The information if the consumer uses Kafka based or zookeeper offsets is > > available to the broker. > > ________________________________________ > > From: Becket Qin <becket....@gmail.com> > > Sent: Sunday, November 13, 2016 4:13:01 AM > > To: dev@kafka.apache.org > > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to > KafkaConsumer > > > > Hi, > > > > We created KIP-92 to propose adding per partition lag metrics to > > KafkaConsumer. > > > > The KIP wiki link is the following: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer > > > > Comments are welcome. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > The information contained in this email is strictly confidential and for > > the use of the addressee only, unless otherwise indicated. If you are not > > the intended recipient, please do not read, copy, use or disclose to > others > > this message or any attachment. Please also notify the sender by replying > > to this email or by telephone (+44(020 7896 0011) and then delete the > email > > and any copies of it. Opinions, conclusion (etc) that do not relate to > the > > official business of this company shall be understood as neither given > nor > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > registered in England and Wales, company number 04008957) and IG Index > > Limited (a company registered in England and Wales, company number > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > > Index Limited (register number 114059) are authorised and regulated by > the > > Financial Conduct Authority. > > >
