Re: [VOTE] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

Hi all, Thanks for voting! KIP-1049 has been accepted with the following +1 votes: - Sophie Blee-Goldman - Matthias J. Sax - Lucas Brutschy - Bruno Cadonna Thanks to everyone who took part in the discussion and voting! I have opened a PR, please review it : https://github.com/apache/kafka

Re: [DISCUSS] Apache Kafka 3.9.0 release

Hi Colin, Just wanted to let you know that KIP-1025 was accepted on July 1st, and the related PR was merged yesterday. Thanks! On Tue, Jul 9, 2024 at 7:47 PM Mickael Maison wrote: > Hi Colin, > > I've added both KIP-1057 (Approved, PR under review) and KIP-1040 > (Approved and PR merged) to th

[jira] [Resolved] (KAFKA-10872) Log broker configuration prefixed with "listener.name.*"

[ https://issues.apache.org/jira/browse/KAFKA-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Badai Aqrandista resolved KAFKA-10872. -- Resolution: Fixed > Log broker configuration prefixed with "listener.name.*" > ---

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

Thanks, I missed that, I can update the KIP tomorrow to include that. Chris On Tue, Jul 9, 2024 at 6:15 PM Chris Egerton wrote: > Hi Chris, > > Thanks for the updates and the additional context. I have one last > nit after doing another readthrough: in the public interfaces section it > states

[jira] [Created] (KAFKA-17105) Unnecessary connector restarts after being newly created

Chris Egerton created KAFKA-17105: - Summary: Unnecessary connector restarts after being newly created Key: KAFKA-17105 URL: https://issues.apache.org/jira/browse/KAFKA-17105 Project: Kafka Is

[jira] [Created] (KAFKA-17104) InvalidMessageCrcRecordsPerSec is not updated in validating LegacyRecord

Chia-Ping Tsai created KAFKA-17104: -- Summary: InvalidMessageCrcRecordsPerSec is not updated in validating LegacyRecord Key: KAFKA-17104 URL: https://issues.apache.org/jira/browse/KAFKA-17104 Project:

Re: [VOTE]: KIP-1032: Upgrade to Jakarta and JavaEE 10 in Kafka 4.0

+1 (binding), thanks for the KIP! On Tue, Jul 9, 2024 at 11:35 AM Greg Harris wrote: > While breaking compatibility is always undesirable, this is unfortunately a > way in which one of our dependencies "leaks through" our API, and makes > this a necessity. > Connect and MirrorMaker2 can follow t

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

Hi Chris, Thanks for the updates and the additional context. I have one last nit after doing another readthrough: in the public interfaces section it states that "There shouldn't be changes to the main client API as the API doesn't use jakarta". This is almost true, but like Greg has mentioned pre

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

hi Michael Thanks for sharing. I have read those two discussion threads. The replicas assignment is a interesting topic to me, so the following questions are used for my education (and learn something from this KIP) Question: 1 When adding brokers to a cluster, Kafka currently does not always p

[jira] [Created] (KAFKA-17103) MockClient tight loops when no metadata is present in KafkaProducerTest

Greg Harris created KAFKA-17103: --- Summary: MockClient tight loops when no metadata is present in KafkaProducerTest Key: KAFKA-17103 URL: https://issues.apache.org/jira/browse/KAFKA-17103 Project: Kafka

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

Hi Chris, I can make those changes. For number 2 that was a holdover back when we were discussing 11 vs 12 and whether or not JDK 17 would be used, but we are going with 12 since 11 is deprecated so I will fix that. On Tue, Jul 9, 2024 at 2:54 PM Chris Egerton wrote: > Hi Chris, > > Thanks for

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

Hi Chris, Thanks for the KIP! It's a bit of a drastic change to rip the bandaid off like this and require users running Connect to upgrade to JDK 17, but I think it's the best out of the options that are available to us. Two small nits on the KIP: 1. It'd be nice to link to KIP-1013 in the motiva

[DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

Hello everyone, I would like to start the discussion thread for KIP-1068. This is a relatively small KIP, only proposing to add a couple of new metrics. If you have any suggestions or feedback, let me know, it will be much appreciated.

Re: [DISCUSS] Apache Kafka 3.8.0 release

Thanks for the update, I'm looking closely to this PR. I'll start working on the first RC shortly after it's merged. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berl

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

Hi Chia-Ping, As mentioned in the rejected alternatives section, the community was against KIP-660. The consensus was to not let users build optimized policy for specific scenarios. The main points were: - the difficulty to expose an extensive view of the cluster state (with KRaft, the cluster met

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

Hi Luke, 2. isCordoned() is a new method on LogDirDescription. It does not take any arguments. It just returns true if this log directory the LogDirDescription represents is cordoned. 3. Sorry that was a typo. This method will only return a log directory that is not cordoned. Fixed 4. Cordoned l

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Chris, As it stands, the error handling for transactions in KafkaProducer is not ideal. There’s no reason why a failed operation should fail a transaction provided that the application can tell that the operation was not included in the transaction and then make its own decision whether to conti

[DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

Hello everyone, I would like to start the discussion thread for KIP-1068. This is a relatively small KIP, only proposing to add a couple of new metrics. If you have any suggestions or feedback, let me know, it will be much appreciated.

Re: [DISCUSS] Apache Kafka 3.8.0 release

The KS system test did surface a bug. Bruno put a PR up for it, which is already approved. We should get it into 3.8 asap to unblock the release from our side. -Matthias On 7/8/24 8:51 AM, Josep Prat wrote: Hi Dongjin, KIP-390 is part of the 3.8 release because the JIRA associated with it:

Re: [DISCUSS] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

Thanks Justine to confirm the rules I have updated the rules to Time Based Release Plan ( https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan), see following: We break compatibility (i.e. remove deprecated public methods after a > reasonable period, and typically wait 1 year

Re: [VOTE]: KIP-1032: Upgrade to Jakarta and JavaEE 10 in Kafka 4.0

While breaking compatibility is always undesirable, this is unfortunately a way in which one of our dependencies "leaks through" our API, and makes this a necessity. Connect and MirrorMaker2 can follow the broker's lead on setting Java 17 as the minimum version. +1 (binding) Thanks! Greg On Tue,

Re: [DISCUSS] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

Hey all, I agree with Chia-Ping that the deprecation should definitely occur in 3.9. My understanding (after discussing offline with some folks) is that we typically wait 1 year after deprecation, but there are some exceptions to that rule. For KIP-1013, we traded off not waiting the whole period

[jira] [Created] (KAFKA-17102) FetchRequest#forgottenTopics would return incorrect data

Chia-Ping Tsai created KAFKA-17102: -- Summary: FetchRequest#forgottenTopics would return incorrect data Key: KAFKA-17102 URL: https://issues.apache.org/jira/browse/KAFKA-17102 Project: Kafka

[jira] [Resolved] (KAFKA-16684) FetchResponse#responseData could return incorrect data

[ https://issues.apache.org/jira/browse/KAFKA-16684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16684. Fix Version/s: 3.9.0 Resolution: Fixed > FetchResponse#responseData could return in

[VOTE]: KIP-1032: Upgrade to Jakarta and JavaEE 10 in Kafka 4.0

Hi Everyone, I'd like to call a vote on https://cwiki.apache.org/confluence/display/KAFKA/KIP-1032%3A+Upgrade+to+Jakarta+and+JavaEE+10+in+Kafka+4.0 This KIP upgrades to the latest Jakarta API and newest version of Jetty 12 and removes the deprecated javax APIs. It also bumps the required version

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

Hi Greg, I will make a couple quick tweaks and open a vote. I think we should target JDK 17, JavaEE 10 and Jetty 12 because Jetty 11 is now EOL. Chris On Mon, Jul 8, 2024 at 2:13 PM Greg Harris wrote: > Hi Chris, > > Please open a vote thread for this. > > Thanks, > Greg > > On Tue, May 14, 20

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

hi Mickael That is totally a good idea, but I have a question about the implementation Do we consider making pluggable ReplicaPlacer (KIP-660) first and then add another impl of ReplicaPlacer to offer cordon mechanism? Noted that `ReplicaPlacer` can implement Reconfigurable to get updated at runt

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

Hi Mickael, Thanks for the KIP! This is a long waiting feature for many users! Questions: 1. I think piggyback the "BrokerHeartbeatRequest" to forward the corden log dir to controller makes sense to me. We already did similar things for fence, controller shutdown, failed log dir...etc. 2. In the

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #3088

See Changes: -- [...truncated 532881 lines...] [2024-07-09T12:41:46.732Z] [2024-07-09T12:41:46.732Z] Gradle Test Run :core:test > Gradle Test Executor 99 > KafkaZkClientT

Re: [DISCUSS] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

Deal all, It seems we need more discussion to reach the consensus on "rules" 1. minor/major release rule 2. 1 year rule the recent adopted KIPs following only rule_1 are shown below: KIP-1041 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=303794933 KIP-1013 https://cwiki.apache

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

Hey, I will open a voting thread by end of this week if there is no more feedbacks. Thanks Omnia > On 2 Jul 2024, at 17:25, Omnia Ibrahim wrote: > > Hi David, Thanks for having a look into this! > >> DJ1: Why do we want to migrate only a subset of the APIs vs migrating all >> of them? For i

[jira] [Resolved] (KAFKA-16741) Add ShareGroupHeartbeat API support in GroupCoordinator

[ https://issues.apache.org/jira/browse/KAFKA-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16741. -- Resolution: Fixed > Add ShareGroupHeartbeat API support in GroupCoordinator >

[jira] [Resolved] (KAFKA-16731) Support for share-group-metrics in the broker

[ https://issues.apache.org/jira/browse/KAFKA-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16731. -- Resolution: Fixed > Support for share-group-metrics in the broker > --

Re: [DISCUSS] Apache Kafka 3.9.0 release

Hi Colin, I've added both KIP-1057 (Approved, PR under review) and KIP-1040 (Approved and PR merged) to the release plan. I also added KIP-1031 as I merged the PR this morning. Thanks, Mickael On Tue, Jul 9, 2024 at 9:22 AM Mario Fiore Vitale wrote: > > Hi Colin, > > Can the KIP-1040[1] be inc

[jira] [Resolved] (KAFKA-16254) Allow MM2 to fully disable offset sync feature

[ https://issues.apache.org/jira/browse/KAFKA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-16254. Resolution: Fixed > Allow MM2 to fully disable offset sync feature > -

Re: [DISCUSS] Apache Kafka 3.9.0 release

Hi Colin, Can the KIP-1040[1] be included? It is accepted and the relative PR has been merged. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1040%3A+Improve+handling+of+nullable+values+in+InsertField%2C+ExtractField%2C+and+other+transformations Thanks, Mario. On Tue, Jul 9, 2024 at 9:18

Re: [DISCUSS] Apache Kafka 3.9.0 release

Hi Colin, thanks, but I don't see my KIP in the release plan. Am I looking at the wrong page? Anyway, I think the PR is ready to be merged, I'm just waiting for one of the reviewers 2nd pass. On Mon, Jul 8, 2024 at 8:07 PM Colin McCabe wrote: > > Hi Federico, > > Thanks for the KIP. I've added it