[DISCUSS] KIP-351: Add --under-min-isr option to describe topics command

Hi All, After some feedback, I have reformulated KIP-351 . This KIP proposes an additional "--under-min-isr" option in TopicCommand to show topic partitions which are under the con

Re: [DISCUSS] KIP-351: Add --under-min-isr option to describe topics command

Bumping this as I have added some additional details. This change will require adding a "--bootstrap-server" flag to identify the current broker/cluster configured "min.insync.replicas". Regards, Kevin On Fri, Oct 12, 2018 at 4:19 PM Kevin Lu wrote: > Hi All, > >

Re: [DISCUSS] KIP-351: Add --under-min-isr option to describe topics command

. In case, the configuration was > specified during creation, the source will be set to > "DYNAMIC_TOPIC_CONFIG". So there's no need to query Zookeeper. > On Fri, Oct 19, 2018 at 5:02 PM Kevin Lu wrote: > > > > Bumping this as I have added some additional detai

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

Hi Viktor, +1 to this KIP. I would very much like to see AdminClient in TopicCommand. This would also allow us to efficiently implement new features like the "--under-min-isr" option I proposed in KIP-351

Re: [DISCUSS] KIP-386: Make Min metrics' default value consistent with Max metrics

Hi Stanislav, Thanks for the KIP! Standardizing this would be extremely helpful. We have been publishing client-side metrics to a time-series database using metrics reporter since 0.8, and we had to do explicit checks like "!Double.isNaN(metricValue)", "metricValue != Double.NEGATIVE_INFINITY", a

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

+1 (non-binding) Regards, Kevin On Thu, Oct 25, 2018 at 8:19 AM Harsha Chintalapani wrote: > Thanks for the KIP. +1 (binding). > -Harsha > On Oct 25, 2018, 10:40 AM -0400, Viktor Somogyi-Vass < > viktorsomo...@gmail.com>, wrote: > > Thanks for the votes so far. > > > > @Colin: yup, I've added t

Re: [DISCUSS] KIP-351: Add --under-min-isr option to describe topics command

Kevin Lu wrote: > Hi Mickael, > > Thanks for the suggestion for getting the "computed" configuration from > AdminClient.describeConfigs(). That's exactly what I was looking for! > > I have updated the KIP to use AdminClient.describeConfigs(), and included >

Re: [DISCUSS] KIP-351: Add --under-min-isr option to describe topics command

e to have and > makes excellent use of the new AdminClient introduced in KIP-377 in my > opinion. > I believe it's perfectly fine to start a vote thread on the fact that > KIP-377 was accepted already. > > On Mon, Nov 5, 2018 at 3:03 PM Kevin Lu wrote: > > > Bu

Re: [VOTE] KIP-386: Standardize on Min/Avg/Max metrics' default values

+1 (non-binding) Regards, Kevin On Tue, Nov 6, 2018 at 1:01 AM Stanislav Kozlovski wrote: > Hey everybody, > > I'm starting a vote thread on KIP-386: Standardize on Min/Avg/Max metrics' > default values > > > In short,

[VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

Hi All, I'm starting the vote thread for KIP-351: Add --under-min-isr option to describe topics command. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command Discussion thread: https://lists.apache.org/thread.html/90d1652ebc03a7b

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

gards, Kevin On Thu, Nov 8, 2018 at 1:07 AM Kevin Lu wrote: > Hi All, > > I'm starting the vote thread for KIP-351: Add --under-min-isr option to > describe topics command. > > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+op

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

binding) > On Thu, Nov 8, 2018 at 1:37 PM Kevin Lu wrote: > > > > Hi All, > > > > I'm starting the vote thread for KIP-351: Add --under-min-isr option to > > describe topics command. > > > > KIP: > > > https://cwiki.apache.org/confluenc

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

t; -Jason > > On Wed, Nov 28, 2018 at 5:26 PM Kevin Lu wrote: > > > @Manikumar I have updated the Compatibility section to note that this > > option is not supported with the deprecated "--zookeeper" option. > > > > We still need 1 more binding vote for

Requesting Permission To Create KIP And Assign JIRAs

Hi All, I would like to start contributing to Kafka but I do not have access to create KIPs or assign JIRA to myself. Can someone set it up for me? Confluence id: lu.kevin Jira username: lu.kevin Email: lu.ke...@berkeley.edu Thanks! Regards, Kevin

Re: Requesting Permission To Create KIP And Assign JIRAs

n 26, 2018 at 10:37 PM Kevin Lu wrote: > Hi All, > > I would like to start contributing to Kafka but I do not have access to > create KIPs or assign JIRA to myself. > > Can someone set it up for me? > > Confluence id: lu.kevin > Jira username: lu.kevin > > Email: lu

[DISCUSS] KIP-344: The auto-generated client id should be passed to MetricsReporter

Hi All, This is the discussion thread for KIP-344 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-344%3A+The+auto-generated+client+id+should+be+passed+to+MetricsReporter ). Thanks! Regards, Kevin

[VOTE] KIP-344: The auto-generated client id should be passed to MetricsReporter

Hi All, I am calling a vote for KIP-344 https://cwiki.apache.org/confluence/display/KAFKA/KIP-344%3A+The+auto-generated+client+id+should+be+passed+to+MetricsReporter . If any committer feels that this minor change does not need to go through the KIP process, then feel free to leave a code review

[DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

Hi friends! This thread is to discuss KIP-351 ! I am proposing to add a --critical-partitions option to the describe topics command that will only list out topic partitions t

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

t; broker metric and it's usually pretty relevant when investigating > issues > > On Thu, Aug 2, 2018 at 8:54 AM, Kevin Lu wrote: > > Hi friends! > > > > This thread is to discuss KIP-351 > > < > https://cwiki.apache.org/confluence/display/KAFKA/K

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

be > indicative of a more severe problem and could be cause for a call from > pagerduty. > > -Jason > > On Thu, Aug 2, 2018 at 9:28 AM, Kevin Lu wrote: > > > Hi Mickael, > > > > Thanks for the suggestion! > > > > Correct me if I am mistaken, bu

Re: [VOTE] KIP-344: The auto-generated client id should be passed to MetricsReporter

Yu wrote: > +1 > > On Wed, Jul 25, 2018 at 9:49 AM Kevin Lu wrote: > > > Hi All, > > > > I am calling a vote for KIP-344 > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-344%3A+The+auto-generated+client+id+should+be+passed+to+Metrics

[DISCUSS] KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metrics

Hi friends! :) I believe we currently have a gap in KafkaConsumer metrics for errors since the KafkaConsumer is complex and are many places where things can go wrong. Currently, these failures are logged and certain ones can be inferred from the existing metrics (ex. heartbeat-rate). This KIP see

Re: [DISCUSS] KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metrics

Bumping this as it has been a week without any responses. Regards, Kevin On Wed, Aug 15, 2018 at 11:40 PM Kevin Lu wrote: > Hi friends! :) > > I believe we currently have a gap in KafkaConsumer metrics for errors > since the KafkaConsumer is complex and are many places where t

Re: [DISCUSS] KIP-363: Allow performance tools to print final results to output file

Hi Attila, Thanks for the KIP. I think this would be a useful feature. Every time I have to benchmark using these performance tools, I end up redirecting the output to a file anyways. Just a couple minor questions... 1. If the configured file already exists, what would be the outcome? My intuit

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

will only have direct overrides, and will not have the broker's configured min.insync.replicas. Any ideas on how to approach this? Regards, Kevin On Mon, Aug 6, 2018 at 8:21 AM Kevin Lu wrote: > Hi Jason, > > Thanks for the response! > > I completely agree with you and

MetricsReporter does not get auto-generated client.id

I have already created a Jira for this, but have not received any comments so bringing this to email. https://issues.apache.org/jira/browse/KAFKA-6123 Interceptors and MetricsReporters configured for a client will receive the user-specified configurations via the configure() method. When a user

[DISCUSS] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

Hi All, Getting the discussion thread started for KIP-427 in case anyone is free right now. I’d like to propose a new category of topic partitions *AtMinIsr* which are partitions that only have the minimum number of in sync replicas left in the ISR set (as configured by min.insync.replicas). Thi

Re: [DISCUSS] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

to drop from 3 to 2 but it can still be > higher than the minIsr (say 1). And a planned maintenance can cause ISR to > drop from 3 to 2, which trigger the AtMinIsr metric if minIsr=2. Can you > update the design doc to fix or explain this issue? > > Thanks, > Dong > &g

Re: [DISCUSS] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

> > this metric. > > > > In the Usage section, it is mentioned that user needs to manually check > > whether there is ongoing maintenance after AtMinIsr is triggered. Could > you > > explain how is this different from the current way where we use > > Un

Re: [VOTE] 2.2.0 RC1

Hi Matthias, The Confluence release page is correct, but the “ - Add --under-min-isr option to describe topics command” was pushed back to 2.3.0. Thanks, Kevin On Fri, Mar 1, 2019 at 11:48 AM Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the second c

[VOTE] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

Hi All, I would like to start the vote thread for KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option). https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089398 Thanks! Regards, Kevin

Re: [VOTE] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

>> --Vahid > > > >> > > > >> On Wed, Mar 6, 2019 at 8:39 PM Dongjin Lee > > wrote: > > > >> > > > >>> +1 (non-binding) > > > >>> > > > >>> On Wed, Mar 6, 2019, 3:14 AM Dong Lin wrote: > > > >

MetricsReporter Does Not Receive Auto-generated Client Id

Hi, Is there a particular reason why the MetricsReporter class's configure() call is not guaranteed to receive the client id? The interceptors' configure() methods are guaranteed to have client id field (either user specified or the kafka auto generated client id such as producer-1). We are impl

[DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

Hi All, I'd like to propose a new consumer metric that measures the time between calls to poll() for use in issues related to hitting max.poll.interval.ms due to long processing time. https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+cal

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

Hi All, Happy Friday! Bumping this. Any thoughts? Thanks. Regards, Kevin On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu wrote: > Hi All, > > I'd like to propose a new consumer metric that measures the time between > calls to poll() for use in issues related to hitting max.poll.int

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

igher the % value the more > > indicative of user code being the cause of performance bottlenecks. > > > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu wrote: > > > > > > Hi All, > > > > > > Happy Friday! Bumping this. Any thoughts? > &

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

nk the metric will be really useful to indicate > the perf bottlenecks on user code vs kakfa consumer/broker. > > Thanks, > Harsha > > > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu wrote: > > > Hi Radai & Jason, > > > > Thanks for the support and suggestion.

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

. > > > > > > On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu wrote: > > > > > Hi Harsha, > > > > > > Thanks for the feedback. I have added *last-poll-seconds-ago* to the > KIP > > > (being consistent with *last-heartbeat-seconds-ago*). > &g

[VOTE] KIP-517: Add consumer metrics to observe user poll behavior

Hi All, Since we have a bit of support, I'd like to start the vote on KIP-517: https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior Thanks! Regards, Kevin

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

hen the application is busy processing, it approaches 0. > Does that make sense? > > Thanks, > Jason > > > On Tue, Sep 17, 2019 at 7:26 PM Satish Duggana > wrote: > > > Hi Kevin, > > Thanks for adding useful metrics with the KIP. > > > > On Wed, 18 S

Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior

; ka...@harsha.io> > > > > > wrote: > > > > > > > > > > > +1 (binding). Thanks for the KIP. > > > > > > > > > > > > -Harsha > > > > > > > > > > >

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

+1 (non-binding) Definitely needed this before as it would have saved me some time from trying to guess a client's version from api version/source code. Thanks for the KIP! Regards, Kevin On Fri, Sep 20, 2019 at 4:29 PM Jason Gustafson wrote: > +1 from me. This is a clever solution. Kind of a

Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior

Hi All, It has been a few days so I am closing the vote now. This KIP has passed with +4 binding votes (Harsha, Manikumar, Jason, Bill) and +3 non-binding votes (Mickael, Maulin, Tom). Regards, Kevin On Fri, Sep 20, 2019 at 1:37 PM Kevin Lu wrote: > Hi All, > > Thanks for the vot

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, Please add KIP-517 as it is now accepted too https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior Thanks. Regards, Kevin On Tue, Sep 24, 2019 at 7:25 AM Manikumar wrote: > Hi All, > > Just a reminder that tomorrow (Sep 2

Re: [ANNOUNCE] New committer: Mickael Maison

Congratulations Mickael! Very well deserved! Regards, Kevin On Thu, Nov 7, 2019 at 5:12 PM Gurudatt Kulkarni wrote: > Congratulations Mickael! > > On Friday, November 8, 2019, Matthias J. Sax > wrote: > > Congrats Mickeal! > > > > > > -Matthias > > > > On 11/7/19 1:53 PM, Bill Bejeck wrote: >

Re: [VOTE] KIP-518: Allow listing consumer groups per state

+1 (non-binding) I can see this being useful. Thanks for the KIP! Regards, Kevin On Mon, Nov 18, 2019 at 4:09 AM Mickael Maison wrote: > Hi all, > > I'd like to start the vote on KIP-518: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state >

[jira] [Created] (KAFKA-7722) Increase ProducerPerformance precision by using nanoTime

Kevin Lu created KAFKA-7722: --- Summary: Increase ProducerPerformance precision by using nanoTime Key: KAFKA-7722 URL: https://issues.apache.org/jira/browse/KAFKA-7722 Project: Kafka Issue Type: New

[jira] [Resolved] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[ https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Lu resolved KAFKA-5991. - Resolution: Fixed > Change Consumer per partition lag metrics to put topic-partition-id in t

[jira] [Created] (KAFKA-7207) Make rate & total metrics documentation consistent

Kevin Lu created KAFKA-7207: --- Summary: Make rate & total metrics documentation consistent Key: KAFKA-7207 URL: https://issues.apache.org/jira/browse/KAFKA-7207 Project: Kafka Issue

[jira] [Created] (KAFKA-7218) Log thread names & ids when KafkaConsumer throws ConcurrentModificationException

Kevin Lu created KAFKA-7218: --- Summary: Log thread names & ids when KafkaConsumer throws ConcurrentModificationException Key: KAFKA-7218 URL: https://issues.apache.org/jira/browse/KAFKA-7218 Project: K

[jira] [Created] (KAFKA-7236) Add --critical-partitions option to describe topics command

Kevin Lu created KAFKA-7236: --- Summary: Add --critical-partitions option to describe topics command Key: KAFKA-7236 URL: https://issues.apache.org/jira/browse/KAFKA-7236 Project: Kafka Issue Type

[jira] [Created] (KAFKA-7300) Add Fetcher fetch-error-rate and fetch-error-total metrics

Kevin Lu created KAFKA-7300: --- Summary: Add Fetcher fetch-error-rate and fetch-error-total metrics Key: KAFKA-7300 URL: https://issues.apache.org/jira/browse/KAFKA-7300 Project: Kafka Issue Type

[jira] [Created] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

Kevin Lu created KAFKA-5991: --- Summary: Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name Key: KAFKA-5991 URL: https://issues.apache.org/jira/browse/KAFKA-5991

[jira] [Created] (KAFKA-6123) MetricsReporter does not get auto-generated client.id

Kevin Lu created KAFKA-6123: --- Summary: MetricsReporter does not get auto-generated client.id Key: KAFKA-6123 URL: https://issues.apache.org/jira/browse/KAFKA-6123 Project: Kafka Issue Type

[jira] [Created] (KAFKA-7904) Add AtMinIsr topic partition category

Kevin Lu created KAFKA-7904: --- Summary: Add AtMinIsr topic partition category Key: KAFKA-7904 URL: https://issues.apache.org/jira/browse/KAFKA-7904 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent

Kevin Lu created KAFKA-5906: --- Summary: Change metric.reporters configuration key to metrics.reporters to be consistent Key: KAFKA-5906 URL: https://issues.apache.org/jira/browse/KAFKA-5906 Project: Kafka

[jira] [Created] (KAFKA-8874) KIP-517: Add consumer metric indicating time between poll calls

Kevin Lu created KAFKA-8874: --- Summary: KIP-517: Add consumer metric indicating time between poll calls Key: KAFKA-8874 URL: https://issues.apache.org/jira/browse/KAFKA-8874 Project: Kafka Issue