Hey Michael, Thanks for the comments. Exposing the lag on the client side may serve some cases a little different from monitoring. For example, one of the use case we have was that an application has some high priority and low priority topics to consume. They want to switch between consuming from high priority and low priority topics alternately based on the lag on the partition. i.e. if the consume has already caught up with the log end offset of a high priority topic, the application will switch to consume from the low priority topics. Otherwise it will continue consuming from the high priority topics. We have seen a few other similar use cases that require a programmatic access to the lag. Although people can always use offsetsForTimes() to get the LEO, but it is more expensive call involving an RPC and is a blocking call.
Thanks, Jiangjie (Becket) Qin On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Should state I have no objections adding this client side, just more a > question to why we don't look and propose to add this broker side also. > > Sent using OWA for iPhone > ________________________________________ > From: Michael Pearce <michael.pea...@ig.com> > Sent: Monday, November 14, 2016 4:58:45 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to > KafkaConsumer > > Why do we not look to expose the lag broker side centrally? > > Eg like burrow. > > From an operations point it's a lot easier to monitor lag centrally than > per application. Also then you'd be able to see lag of consumers not alive > or stalled. > > The information if the consumer uses Kafka based or zookeeper offsets is > available to the broker. > ________________________________________ > From: Becket Qin <becket....@gmail.com> > Sent: Sunday, November 13, 2016 4:13:01 AM > To: dev@kafka.apache.org > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer > > Hi, > > We created KIP-92 to propose adding per partition lag metrics to > KafkaConsumer. > > The KIP wiki link is the following: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer > > Comments are welcome. > > Thanks, > > Jiangjie (Becket) Qin > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the email > and any copies of it. Opinions, conclusion (etc) that do not relate to the > official business of this company shall be understood as neither given nor > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by the > Financial Conduct Authority. >
