It is incredibly weird, but it is also a repeating pattern. I figured we may as well address this. I definitely explained how Kafka is different from traditional queues and what lag means vs partition size.
To the best of my understanding, they see size as a measure of a topic importance in terms that are easier to their business management to understand. If I store many events in Kafka, clearly I'm making good use of Kafka. "My manager comes every day and asks me, how many events do we keep in Kafka" is a quote I heard multiple times. Those are companies who are new to the whole streams concept... I guess they are not used to think in terms of throughput? And management seems to learn slower than engineers? Anyway, this is just a suggestion. I'll +1 the KIP either way. But since I know there's demand, I figured I'll ask. Gwen On Wed, Jun 27, 2018 at 4:53 PM, Jason Gustafson <ja...@confluent.io> wrote: > Hey Gwen, > > Why do users want partition size? It seems like a weird thing to be > concerned about. Perhaps they are trying to get a sense of the lag as a > percentage of the total size of the partition or something like that? > > -Jason > > On Tue, Jun 26, 2018 at 9:12 PM, Gwen Shapira <g...@confluent.io> wrote: > > > Thank you! > > > > On Tue, Jun 26, 2018 at 8:47 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com> wrote: > > > > > Thanks for the feedback. The KIP is updated to also include a > "partition > > > size" column. > > > > > > --Vahid > > > > > > > > > > > > > > > From: Ted Yu <yuzhih...@gmail.com> > > > To: dev@kafka.apache.org > > > Date: 06/26/2018 06:21 PM > > > Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to > > > Show Beginning Offsets > > > > > > > > > > > > nit: > > > > > > bq. leaving this empty for compacted topics > > > > > > Some user(s) may be confused by empty partition size. How about > emitting > > > 'compacted' for compacted topics ? > > > > > > Cheers > > > > > > On Tue, Jun 26, 2018 at 4:42 PM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > > > It will be. In my experience most topics aren't compacted, so it will > > > still > > > > be valuable. If not difficult, leaving this empty for compacted > topics > > > to > > > > avoid confusion will also be nice. > > > > > > > > On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian < > > > > vahidhashem...@us.ibm.com> wrote: > > > > > > > > > Hi Gwen, > > > > > > > > > > Thanks for the feedback. > > > > > Regarding the partition size, couldn't "end offset - start offset" > be > > > > > misleading for compacted topics? > > > > > > > > > > --Vahid > > > > > > > > > > > > > > > > > > > > > > > > > From: Gwen Shapira <g...@confluent.io> > > > > > To: dev <dev@kafka.apache.org> > > > > > Date: 06/26/2018 02:36 PM > > > > > Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group > Command > > > to > > > > > Show Beginning Offsets > > > > > > > > > > > > > > > > > > > > Small suggestion: you can also add a "partition size" column - > > > difference > > > > > between log-end and log-start. We've had users ask for this. > > > > > > > > > > On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira <g...@confluent.io> > > > wrote: > > > > > > > > > > > This will be useful! Thank you :) > > > > > > > > > > > > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian < > > > > > > vahidhashem...@us.ibm.com> wrote: > > > > > > > > > > > >> Hi everyone, > > > > > >> > > > > > >> I have created a trivial KIP to improve the offset reporting of > > the > > > > > >> consumer group command: > > > > > >> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-325% > > > > > > > > > > > > > >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets > > > > > >> Looking forward to your feedback! > > > > > >> > > > > > >> Thanks. > > > > > >> --Vahid > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > *Gwen Shapira* > > > > > > Product Manager | Confluent > > > > > > 650.450.2760 | @gwenshap > > > > > > Follow us: Twitter < > > > > > > > > https://twitter.com/ConfluentInc > > > > > > > > > | blog > > > > > > < > > > > > > > > http://www.confluent.io/blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Gwen Shapira* > > > > > Product Manager | Confluent > > > > > 650.450.2760 | @gwenshap > > > > > Follow us: Twitter < > > > > > > > > https://twitter.com/ConfluentInc > > > > > > > > > | blog > > > > > < > > > > > > > > http://www.confluent.io/blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Gwen Shapira* > > > > Product Manager | Confluent > > > > 650.450.2760 | @gwenshap > > > > Follow us: Twitter < > > > https://twitter.com/ConfluentInc > > > > | blog > > > > < > > > http://www.confluent.io/blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > *Gwen Shapira* > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > <http://www.confluent.io/blog> > > > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
