Re: Contributor Permission

Hi Jack, I've added you into the contribution list. Thanks for the interest in Apache Kafka. Luke On Thu, May 12, 2022 at 8:44 AM Jack Burgoyne wrote: > Hello, > > I'd like to start contributing to Kafka, could I have contributor > permission for my Jira username: jcbu2718? > > Thank you, > Ja

Re: Request to be added to Kafka Jira project

Thanks for your help! On Thu, May 12, 2022 at 7:36 AM Luke Chen wrote: > Hi Lim, > > I've added you into the contribution list in JIRA. > Thanks for the interest in Apache Kafka. > > Luke > > On Thu, May 12, 2022 at 2:09 PM Lim Qingwei wrote: > > > Sorry, I misread, my jira id is qingwei91 > >

Re: Request to be added to Kafka Jira project

Hi Lim, I've added you into the contribution list in JIRA. Thanks for the interest in Apache Kafka. Luke On Thu, May 12, 2022 at 2:09 PM Lim Qingwei wrote: > Sorry, I misread, my jira id is qingwei91 > > On Thu, May 12, 2022 at 7:01 AM Lim Qingwei wrote: > > > Hi, I am looking to pick this up

Re: Request to be added to Kafka Jira project

Sorry, I misread, my jira id is qingwei91 On Thu, May 12, 2022 at 7:01 AM Lim Qingwei wrote: > Hi, I am looking to pick this up: > https://issues.apache.org/jira/browse/KAFKA-13882 > > On Wed, May 11, 2022 at 10:04 PM Mickael Maison > wrote: > >> Hi, >> >> Welcome and thanks for your interest i

Re: Request to be added to Kafka Jira project

Hi, I am looking to pick this up: https://issues.apache.org/jira/browse/KAFKA-13882 On Wed, May 11, 2022 at 10:04 PM Mickael Maison wrote: > Hi, > > Welcome and thanks for your interest in Kafka! > What is your JIRA id? > > Thanks, > Mickael > > On Wed, May 11, 2022 at 10:45 PM Lim Qingwei wrot

Re: [VOTE] 3.1.1 RC1

Thank you for sharing your assessment, Tom and Chris. Apache Spark community had a discussion thread on March. https://lists.apache.org/thread/yq6f5gv7gtdbk1ynwpd9hnc547bgz03m "[DISCUSS] Migration guide on upgrading Kafka to 3.1 in Spark 3.3" As a person who initiated that upgrade via SPARK-3683

[jira] [Resolved] (KAFKA-13763) Improve unit testing coverage and flexibility for IncrementalCooperativeAssignor

[ https://issues.apache.org/jira/browse/KAFKA-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-13763. --- Resolution: Done > Improve unit testing coverage and flexibility for > IncrementalCooperati

Re: [VOTE] 3.1.1 RC1

It's worth noting that one of the most frequent offenders (RebalanceSourceConnectorsIntegrationTest.testDeleteConnector), which failed in five of the nine runs from 111 through 119, is actually disabled on 3.2 and trunk as it's caused by a known and not-yet-addressed bug in rebalancing logic. Since

Kafka Consumer Partition Non Uniform Data Fetch/Pull

Hi, We are using open source Confluent Kafka deployed in Kubernetes. We have less consumers than the partitions and as such we have multiple Kafka partitions assigned to each consumer pod. We observed some of these partitions have higher lag than the other, it seems the consumer is prioritizing som

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Hi Francois, This is a bit of an unusual situation. I think a separate KIP may be the safest route here since this one has already gotten the votes necessary to be included in the next release (as long as it can be implemented and reviewed in time), but it'd probably be best if only one of this KI

Contributor Permission

Hello, I'd like to start contributing to Kafka, could I have contributor permission for my Jira username: jcbu2718? Thank you, Jack

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

Thanks for all of the feedback. Some comments below: Luke wrote: > 1. Jason has asked but you didn't answer: What is the default value for ` > metadata.monitor.write.interval.ms`? Thanks for asking again. Looks like I missed this in my previous reply. In the implementation I am currently working

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #50

See

[jira] [Created] (KAFKA-13894) Extend Kafka kerberos auth support to beyond only hostname

Yiming Zang created KAFKA-13894: --- Summary: Extend Kafka kerberos auth support to beyond only hostname Key: KAFKA-13894 URL: https://issues.apache.org/jira/browse/KAFKA-13894 Project: Kafka Issu

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Colin, I don't mind the idea of MM2 users implementing the AdminClient interface. However, there're two disadvantages to this. 1. Having around 70 methods definitions to have "NotImplemented" is one downside, and keep up with these if the AdminClient interface changes. 2. It makes it h

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #924

See

Re: [DISCUSS] Apache Kafka 3.3.0 Release

Hi José, Could we add KIP-618 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors) to the release plan? It's been blocked on review for almost a year now. If we can't make time to review it before the 3.3.0 release I'll probably close my PRs an

Re: Request to be added to Kafka Jira project

Hi, Welcome and thanks for your interest in Kafka! What is your JIRA id? Thanks, Mickael On Wed, May 11, 2022 at 10:45 PM Lim Qingwei wrote: > > Hi Dev at kafka, > > I wish to try to contribute to the project, can I be added to Kafka Jira > project so that I can pick up ticket? > > I am followi

Request to be added to Kafka Jira project

Hi Dev at kafka, I wish to try to contribute to the project, can I be added to Kafka Jira project so that I can pick up ticket? I am following this guide: https://kafka.apache.org/contributing I wish to pick up ticket from this list: https://issues.apache.org/jira/browse/KAFKA-13882?jql=project%

Re: [DISCUSS] Apache Kafka 3.3.0 Release

Great. I went ahead and created a release page for 3.3.0: https://cwiki.apache.org/confluence/x/-xahD The planned KIP content is based on the list of KIPs targeting 3.3.0 in https://cwiki.apache.org/confluence/x/4QwIAw. Please take a look at the list and let me know if I missed your KIP. The 3.3

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

> Stretch clusters are possible with the KRaft architecture. For example, if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1, you could put a KRaft controller node in each region. This is similar to how with ZK you'd put a ZK node in each region. Ah ha! TIL about process.role

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

On Thu, May 5, 2022, at 15:57, Israel Ekpo wrote: > Thanks Colin. > > I think we may need to update the KIP name to reflect the intent of the KIP > and convey everything it’s about if all the 3 action items will be covered > by the same KIP > > It contains three parts: > > Marking KRraft as Product

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

Hi Andrew, Stretch clusters are possible with the KRaft architecture. For example, if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1, you could put a KRaft controller node in each region. This is similar to how with ZK you'd put a ZK node in each region. best, Colin On

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia Ibrahim, I'm sorry, but I am -1 on adding competing Admin interfaces. This would create confusion and a heavier maintenance burden for the project. Since the org.apache.kafka.clients.admin.Admin interface is a Java interface, any third-party software that wants to insert its own implem

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

Thanks, Niket. I also agree with Jason that this is a public API despite the lack of command-line tool, so we do indeed need a KIP. :) One minor point: I suspect that whatever we end up naming the additional fields here, should also be the name of the metrics in KIP-835. So if we go with a metr

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

Hi José, Thanks for the KIP! I think this will be a nice improvement. I had the same question as Luke and Jason: what's the default here for the NoOpRecord time? :) We should add a value here even if we think we'll adjust it later, just to give a feeling for how much traffic this would create.

Re: [DISCUSS] Apache Kafka 3.3.0 Release

Thanks for volunteering to run the release José! +1 -Bill On Wed, May 11, 2022 at 3:26 AM Bruno Cadonna wrote: > Thanks José! > > +1 > > Best, > Bruno > > On 11.05.22 04:38, John Roesler wrote: > > +1 from me! Thanks for volunteering, José. > > -John > > > > On Tue, May 10, 2022, at 17:53, Jos

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

José, thanks for the KIP! I think this is a good approach for proving the liveness of the quorum when metadata is not changing. 1. Based on the config name "metadata.monitor.write.interval.ms" I'm guessing the intention is to have a regularly scheduled write. If the quorum is busy with lots of the

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #49

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #923

See

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

Thanks Jim, great discussion. +1 from me (non-binding) Cheers, Leah On Wed, May 11, 2022 at 10:14 AM Bill Bejeck wrote: > Thanks for the KIP! > > +1 (binding) > > -Bill > > On Wed, May 11, 2022 at 9:36 AM Luke Chen wrote: > > > Hi Jim, > > > > I'm +1. (please add some note in KIP about the str

[jira] [Resolved] (KAFKA-13893) Add BrokerApiVersions Api in AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-13893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengziming resolved KAFKA-13893. Resolution: Invalid > Add BrokerApiVersions Api in AdminClient > -

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Hi Luke, John, Thanks for bringing up this and also sorting it out! I have added a note to the KIP. Thanks, Jim On Wed, May 11, 2022 at 9:34 AM Luke Chen wrote: > Thanks John! > It makes sense. > I have no other questions as long as it is documented in the KIP. > > Thank you. > Luke > > On W

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

Thanks for the KIP! +1 (binding) -Bill On Wed, May 11, 2022 at 9:36 AM Luke Chen wrote: > Hi Jim, > > I'm +1. (please add some note in KIP about the stream resetting tool can't > be used in paused state) > Thanks for the KIP! > > Luke > > On Wed, May 11, 2022 at 9:09 AM Guozhang Wang wrote: >

[jira] [Created] (KAFKA-13893) Add BrokerApiVersions Api in AdminClient

dengziming created KAFKA-13893: -- Summary: Add BrokerApiVersions Api in AdminClient Key: KAFKA-13893 URL: https://issues.apache.org/jira/browse/KAFKA-13893 Project: Kafka Issue Type: Task

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

Hi Jim, I'm +1. (please add some note in KIP about the stream resetting tool can't be used in paused state) Thanks for the KIP! Luke On Wed, May 11, 2022 at 9:09 AM Guozhang Wang wrote: > Thanks Jim. +1 from me. > > On Tue, May 10, 2022 at 4:51 PM Matthias J. Sax wrote: > > > I had one minor

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Thanks John! It makes sense. I have no other questions as long as it is documented in the KIP. Thank you. Luke On Wed, May 11, 2022 at 9:15 PM John Roesler wrote: > Hi Luke, > > It’s not my KIP, but my two cents is that users should not run the reset > tool while the application is paused. > >

Re: [HEADS-UP] Modification to KIP Template

+1 from me also. This is a great idea. Thanks, John On Wed, May 11, 2022, at 02:49, Luke Chen wrote: > Hi Mickael, > > +1 to add "test plan" section to KIP template. > Thanks for the improvement! > > Luke > > On Tue, May 10, 2022 at 11:21 PM Mickael Maison > wrote: > >> Hi, >> >> I did not see

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Hi Luke, It’s not my KIP, but my two cents is that users should not run the reset tool while the application is paused. The reset tool should only be run while the whole app is shut down because it messes with a lot of internal state bits without synchronization. Leaving the app running (even

Re: [DISCUSS] KIP-781: Allow MirrorMaker 2 to override the client configurations

Hi Dongjin, nice spot for this bug. I wonder if this really needs a KIP as it seems like a clear bug in MM2's interpretation for the config. If you're going to keep the KIP can you please add some details in the Public interface or Proposed changes to list which part of MM2 codebase will change to

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #122

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #121

See

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

To be clear, there is no problem for me to update the current KIP with the builder approach. It's not a lot of work in terms of code. So, up to you. Let me know and I will do the necessary to go in one or the other direction... Thanks again for the feedbacks. Le mer. 11 mai 2022 à 10:52, François

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Hello, Builder is clearly the way to go for future releases of Kafka. If we align streams, we would have 3 builders new ConsumerBuilder() .withKeyDeserializer() .withValueDeserializer() .withInterceptors() .withMetricsReporter() .build(); new ProducerBuilder() .withKeySerializ

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

> And if people start using RemainingLogs and RemainingSegments and then REALLY FEEL like they need RemainingBytes, then we can always add it in the future. +1 Thanks James! Luke On Wed, May 11, 2022 at 3:57 PM James Cheng wrote: > Hi Luke, > > Thanks for the detailed explanation. I agree that

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

Hi Luke, Thanks for the detailed explanation. I agree that the current proposal of RemainingLogs and RemainingSegments will greatly improve the situation, and that we can go ahead with the KIP as is. If RemainingBytes were straight-forward to implement, then I’d like to have it. But we can liv

Re: [HEADS-UP] Modification to KIP Template

Hi Mickael, +1 to add "test plan" section to KIP template. Thanks for the improvement! Luke On Tue, May 10, 2022 at 11:21 PM Mickael Maison wrote: > Hi, > > I did not see any comments nor concerns so I went ahead and added the > Test Plan section to the KIP template. > > Thanks, > Mickael > >

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Hi Francois and Chris, I find the idea of the builder interesting. I think we should go ahead with the current KIP as it is to allow Francois to fix his issue soon. If one of you or both want to push forward the builder idea, feel free to create a new KIP and discuss it with the community.

Re: [VOTE] 3.1.1 RC1

Hi Dongjoon, I've been trying to get a green build of the 3.1 branch from Jenkins ( https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my original email promised to provide this. Individually each stage between builds 111 and 119 has passed in at least one run, but there have been no

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Hi Jim, Thanks for the KIP. Overall LGTM! One late question: Could we run the stream resetter tool (i.e. kafka-streams-application-reset.sh) during pause state? I can imagine there's a use case that after pausing for a while, user just want to continue with the latest offset, and skipping the int

Re: [DISCUSS] Apache Kafka 3.3.0 Release

Thanks José! +1 Best, Bruno On 11.05.22 04:38, John Roesler wrote: +1 from me! Thanks for volunteering, José. -John On Tue, May 10, 2022, at 17:53, José Armando García Sancio wrote: Hi all, I would like to volunteer for the release of Apache Kafka 3.3.0. If people agree, I'll start working

Re: [VOTE] 3.1.1 RC1

Hi, Tom. Could you conclude this vote as a release manager? :) Dongjoon. On 2022/05/06 13:31:15 Michal Tóth wrote: > Hello > > I have executed some produce/consume system tests which all passed. > Also everything passed from https://github.com/tombentley/kafka-verify-rc > - checking signatures,