Any additional feedback on whether we should also include a partition size column or not?
Options: 1. The current KIP (with a partition size column): https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets+and+Partition+Size ** Requires additional processing to extract topic configs ** Will not return a precise size for compacted topics ** Simplifies checking the consumption progress on each partition 2. The earlier version (without a partition size column): https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87296412 ** Less processing overhead compared to Option 1 ** Partition size can be provided via kafka-topics tool For further info, please see the recent discussions on the thread. Thanks! --Vahid From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: dev@kafka.apache.org Date: 07/03/2018 09:11 AM Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets Hi Jason, Thanks for the feedback. Your suggestions make sense to me. I think I'm more in favor of adding this info to kafka-topic tool (through another KIP) since it is not consumer group specific. I'll wait for Gwen and others to comment before making changes to the KIP. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev <dev@kafka.apache.org> Date: 06/28/2018 02:39 PM Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets Hey Gwen/Vahid, I think that use case makes sense, but isn't it a little odd to go through the consumer group tool? I would expect to find something like that from the kafka-topics tool instead. I don't feel too strongly about it, but I hate to complicate the tool by adding the need to query topic configs. If we don't have a meaningful number to report for compacted topics anyway, then it feels like only a half solution. I'd probably suggest leaving this out or just reporting the absolute difference even if a topic is compacted. -Jason On Thu, Jun 28, 2018 at 1:05 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi James, > > > > Thanks for the feedback. I updated the KIP and added some of the benefits > > of this improvement (including some that you mentioned). > > > > Regards. > > --Vahid > > > > > > > > From: James Cheng <wushuja...@gmail.com> > > To: dev@kafka.apache.org > > Date: 06/27/2018 09:13 PM > > Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to > > Show Beginning Offsets > > > > > > > > The “Motivation” section of the KIP says that the starting offset will be > > useful but doesn’t say why. Can you add a use-case or two to describe how > > it will be useful? > > > > In our case (see > > https://github.com/wushujames/kafka-utilities/blob/master/Co > nsumerGroupLag/README.md > > ), we found the starting offset useful so that we could calculate > > partition size so that we could identify empty partitions (partitions > > where all the data had expired). In particular, we needed that info so > > that we could calculate “lag”. Consider that case where we are asked to > > mirror an abandoned topic where startOffset==endOffset==1000000. We would > > have no committed offset on it, and the topic has no data in it, so we > > won’t soon get any committed offset, and so “lag” is kind of undefined. We > > used the additional startOffset to allow us to detect this case. > > > > -James > > > > Sent from my iPhone > > > > > On Jun 26, 2018, at 11:23 AM, Vahid S Hashemian > > <vahidhashem...@us.ibm.com> wrote: > > > > > > Hi everyone, > > > > > > I have created a trivial KIP to improve the offset reporting of the > > > consumer group command: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-325% > 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets > > > > > Looking forward to your feedback! > > > > > > Thanks. > > > --Vahid > > > > > > > > > > > > > > > >
