Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Vahid, Thank you so much for the review and voting! Best, Qichao On Wed, Nov 29, 2023 at 2:55 PM Vahid Hashemian wrote: > Hi Qichao, > > Thanks for answering my questions and updating the KIP accordingly. > > It looks good to me. > > --Vahid > > > On Tue, Nov 28, 2023, 7:22 PM Qichao Chu w

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for answering my questions and updating the KIP accordingly. It looks good to me. --Vahid On Tue, Nov 28, 2023, 7:22 PM Qichao Chu wrote: > Hi Vahid, > > Thank you for taking the time to review the KIP and asking great questions. > > The execution path mentioned is exactly

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Vahid, Thank you for taking the time to review the KIP and asking great questions. The execution path mentioned is exactly how this KIP is going to function. We believe this config, compared with many other configurations like quota, will not pose significant alteration to the broker's functio

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for proposing this KIP. It'd be super valuable to have the ability to have those partition level metrics for Kafka topics. Sorry I'm late to the discussion. I just wanted to bring up a point for clarification and one question: Let's assume that a production cluster cannot affor

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Thank you for updating the KIP Qichao. I don't have any more questions or suggestions. Looks good to move forward from my perspective. -- Divij Vaidya On Fri, Nov 10, 2023 at 2:25 PM Qichao Chu wrote: > Thank you again for the nice suggestions, Jorge! > I will wait for Divij's response and

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Thank you again for the nice suggestions, Jorge! I will wait for Divij's response and move it to the vote stage once the generic filter part reached concensus. Qichao Chu Software Engineer | Data - Kafka [image: Uber] On Fri, Nov 10, 2023 at 6:49 AM Jorge Esteban Quilcate Oto

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for updating the KIP, all updates look good to me. Looking forward to see this KIP moving forward! Cheers, Jorge. On Wed, 8 Nov 2023 at 08:55, Qichao Chu wrote: > Hi Divij, > > Thank you for the feedback. I updated the KIP to make it a little bit more > generic: filters wi

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Divij, Thank you for the feedback. I updated the KIP to make it a little bit more generic: filters will stay in an array instead of different top-level objects. In this way, if we need language filters in the future. The logic relationship of filters is also added. Hi Jorge, Thank you for the

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for the KIP! This will be a valuable contribution and improve the tooling for troubleshooting. I have a couple of comments: 1. It's unclear from the `metrics.verbosity` description what the supported values are. In the description mentions "If the value is high ... In the low s

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Thanks again for making the changes. 1) sounds good 3) sounds good For 2) I was recommending something different where we have a schema such that we can add a list consisting of generic filters but we will start with implementing a singleton list containing only topic filter. I need some more tim

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Divij, It would be very nice if you could take a look at the recent changes, thank you! If there's no more required changes, shall we move to vote stage? Best, Qichao Chu Software Engineer | Data - Kafka [image: Uber] On Thu, Nov 2, 2023 at 12:06 AM Qichao Chu wrote: >

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Divij, Thank you for the very quick response and the nice suggestions. I have updated the KIP with the following thoughts. 1. I checked the Java documentation and it seems the regex engine in utils is n

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Thank you for making the changes Qichao. We are now entering in the territory of defining a declarative schema for filters. In the new input format, the type is string but we are imposing a schema for the string and we should clearly call out the schema. You can perhaps choose to adopt a schema su

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Divij, Thank you for the review and the great suggestions, again. I have updated the corresponding content, can you take another look? Regarding the KIP-544 style regex, I have added it to the new property too. It's expanded to include multiple sections for better future extension. Best, Qicha

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hey *Qichao* Thank you for the update on the KIP. I like the idea of incremental delivery and adding which metrics support this verbosity as a later KIP. But I also want to ensure that we wouldn't have to change the current config when adding that in future. Hence, we need some discussion on it in

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hello Everyone, Can I ask for some feedback regarding KIP-977 ? Best, Qichao Chu Software Engineer | Data - Kafka [image: Uber] On Mon, Oct 16, 2023 at 7:34 PM Qichao Chu wrote

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Divij and Kirk, Thank you both for providing the valuable feedback and sorry for the delay. I have just updated the KIP to address the comments. 1. Instead of using a topic-level control, global verbosity control makes more sense if we want to extend it in the future. It would be very

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Oh, and does metrics.partition.level.reporting.topics allow for regex? > On Sep 12, 2023, at 8:26 AM, Kirk True wrote: > > Hi Qichao, > > Thanks for the KIP! > > Divij—questions/comments inline... > >> On Sep 11, 2023, at 4:32 AM, Divij Vaidya wrote: >> >> Thank you for the proposal Qichao.

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Hi Qichao, Thanks for the KIP! Divij—questions/comments inline... > On Sep 11, 2023, at 4:32 AM, Divij Vaidya wrote: > > Thank you for the proposal Qichao. > > I agree with the motivation here and understand the tradeoff here > between observability vs. increased metric dimensions (metric fan

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

Thank you for the proposal Qichao. I agree with the motivation here and understand the tradeoff here between observability vs. increased metric dimensions (metric fan-out as you say in the KIP). High level comments: 1. I would urge you to consider the extensibility of the proposal for other type