Re: New release branch 3.9

I would recommend against large refactorings in trunk until the first RC for 3.9 - that will reduce cherry-pick friction. Once we have the first RC, subsequent changes to 3.9 should be limited in scope. Ismael On Tue, Jul 30, 2024 at 4:31 PM Colin McCabe wrote: > Yeah, please go ahead. I know a

[jira] [Resolved] (KAFKA-17205) Allow topic config validation in controller level in KRaft mode

[ https://issues.apache.org/jira/browse/KAFKA-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-17205. --- Resolution: Fixed > Allow topic config validation in controller level in KRaft mode > --

Re: [VOTE] KIP-1070: Deprecate MockProcessorContext

Hi, Also a +1 (binging) from me. I am closing this vote with the KIP being accepted with 4 binding +1s from Sophie, Bill, Lucas, and myself. Thanks a lot! -Matthias On 7/26/24 10:52 AM, Matthias J. Sax wrote: I updated the KIP accordingly. If there is no further follow usp, I'll close

Re: [DISCUSS] KIP 1072 - Add @FunctionalInterface annotation to Kafka Streams SAM methods

Thanks a lot for the KIP Ray! It seems to be a good improvement to make using KS with Clojure more seamless. However, I am not 100% sure if all listed interfaces make sense? (100) GlobalKTable: it's basically a sibling to `KStream`, and `KTable` interfaces, but users would never implemented

[jira] [Created] (KAFKA-17224) Make ForeachProcessor internal

Matthias J. Sax created KAFKA-17224: --- Summary: Make ForeachProcessor internal Key: KAFKA-17224 URL: https://issues.apache.org/jira/browse/KAFKA-17224 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Thanks for updating the KIP. (100) As stated previously, I personally don't think that adding Headers to RecordMetadata is the right thing to do. To me, Headers store "application metadata" of a record, but they are not "Kafka native" record metadata (ie metadata Kafka an reason about). Header

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thanks for running the release, and thanks everyone contributed to v3.8.0. On Wed, Jul 31, 2024 at 6:09 AM Greg Harris wrote: > Thank you to all of the Contributors, Committers, and our release manager > Josep! > > Greg > > On Tue, Jul 30, 2024 at 1:34 PM Justine Olshan > > wrote: > > > Thank

Re: New release branch 3.9

Hi Colin and all, If KIP-853 can complete in v3.9.0 in time (or a little delay), I agree we should try to keep v3.9.0 as the last release before v4.0. This way, it will let all Kafka ecosystem projects have a clear (and certain) picture about what will happen in Apache Kafka. Hi Colin, For KIP-95

[jira] [Resolved] (KAFKA-16346) Fix flaky MetricsTest.testMetrics

[ https://issues.apache.org/jira/browse/KAFKA-16346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16346. Fix Version/s: 4.0.0 Resolution: Fixed > Fix flaky MetricsTest.testMetrics > --

[jira] [Resolved] (KAFKA-12432) Fix AdminClient timeout handling in the presence of badly behaved brokers

[ https://issues.apache.org/jira/browse/KAFKA-12432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-12432. Fix Version/s: 3.0.0 Resolution: Fixed > Fix AdminClient timeout handling in the pr

[jira] [Created] (KAFKA-17223) Retrying the call after encoutering UnsupportedVersionException will cause ConcurrentModificationException

Chia-Ping Tsai created KAFKA-17223: -- Summary: Retrying the call after encoutering UnsupportedVersionException will cause ConcurrentModificationException Key: KAFKA-17223 URL: https://issues.apache.org/jira/browse

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3153

See

Re: New release branch 3.9

Yeah, please go ahead. I know a lot of people are waiting for 4.0. best, Colin On Tue, Jul 30, 2024, at 16:05, Matthias J. Sax wrote: > Thanks for clarifying Colin. So my assumptions were actually correct. > > We have a lot of contributors waiting to pick-up 4.0 tickets, and I'll > go ahead a t

[jira] [Created] (KAFKA-17222) Remove the subclass of KafkaMetricsGroup

Chia-Ping Tsai created KAFKA-17222: -- Summary: Remove the subclass of KafkaMetricsGroup Key: KAFKA-17222 URL: https://issues.apache.org/jira/browse/KAFKA-17222 Project: Kafka Issue Type: Impr

Re: New release branch 3.9

Thanks for clarifying Colin. So my assumptions were actually correct. We have a lot of contributors waiting to pick-up 4.0 tickets, and I'll go ahead a tell them that we are ready and they can start to pick them up. Thanks. -Matthias On 7/30/24 3:51 PM, Colin McCabe wrote: Hi Chia-Ping Tsa

Re: New release branch 3.9

Hi Igor, Correct. The whole point of 3.9 is to complete KIP-853. We had many email threads on this :) As release manager I have discretion to delay the release until this is complete. And I will do that. But I do not think the delay will be more than a few days, based on what we're looking at

Re: New release branch 3.9

Hi Chia-Ping Tsai, If you can get them done this week then I think we can merge them in to 3.9. If not, then let's wait until 4.0, please. best, Colin On Tue, Jul 30, 2024, at 09:07, Chia-Ping Tsai wrote: > hi Colin, > > Could you please consider adding > https://issues.apache.org/jira/browse/

Re: New release branch 3.9

On Tue, Jul 30, 2024, at 15:22, Matthias J. Sax wrote: > Thanks Greg. Overall, this makes sense to me. > > The only assumption on my side was, that we are actually pretty sure > that we will hit case (1) or (2)... > > And I actually also thought, that completing the required KRaft work > would on

Re: New release branch 3.9

On Tue, Jul 30, 2024, at 15:35, Greg Harris wrote: > Hi Matthias, > > I agree with you. > >> The only assumption on my side was, that we are actually pretty sure >> that we will hit case (1) or (2)... > > I want this to be the case, but we were pretty sure that KIP-853 was going > to be in 3.8 up u

Re: New release branch 3.9

On Tue, Jul 30, 2024, at 09:52, Matthias J. Sax wrote: > Thanks for cutting the release branch. > > It's great to see `trunk` being bumped to 4.0-SNAPSHOT, and I wanted to > follow up on this: > > We have a bunch of tickets that we can only ship with 4.0 release, and > these tickets were blocked

Re: New release branch 3.9

On Tue, Jul 30, 2024, at 08:59, José Armando García Sancio wrote: > Thanks Colin. > > For KIP-853 (KRaft Controller Membership Changes), we still have the > following features that are in progress. > > 1. UpdateVoter RPC and request handling > > 2

Re: New release branch 3.9

Hi all, KIP-853 is shipping in 3.9. If we have to delay 3.9 to accomplish this, we will, but that seems very unlikely at this point. We are mostly on schedule so far. Trunk is 4.0. best, Colin On Tue, Jul 30, 2024, at 15:35, Greg Harris wrote: > Hi Matthias, > > I agree with you. > >> The on

Re: New release branch 3.9

Hi Matthias, I agree with you. > The only assumption on my side was, that we are actually pretty sure > that we will hit case (1) or (2)... I want this to be the case, but we were pretty sure that KIP-853 was going to be in 3.8 up until it wasn't ready. I believe we need to plan for the worst ca

Re: New release branch 3.9

Thanks Greg. Overall, this makes sense to me. The only assumption on my side was, that we are actually pretty sure that we will hit case (1) or (2)... And I actually also thought, that completing the required KRaft work would only take a few more weeks, and that is also why we have a 3.9 rel

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thank you to all of the Contributors, Committers, and our release manager Josep! Greg On Tue, Jul 30, 2024 at 1:34 PM Justine Olshan wrote: > Thanks Josep for your hard work! And to everyone who contributed to this > release. > > Justine > > On Tue, Jul 30, 2024 at 8:04 AM Kamal Chandraprakash

Re: [VOTE] KIP-1022 Formatting and Updating Features

Thanks for updating the KIP, Justine. Jun On Tue, Jul 30, 2024 at 1:37 PM Justine Olshan wrote: > I added this update to the end of the section Colin added. > > Justine > > On Tue, Jul 30, 2024 at 11:01 AM Jun Rao wrote: > > > Hi, Colin, > > > > Thanks for the update. We also excluded supporte

Re: New release branch 3.9

Hi all, I'd like to clarify my understanding of the path forward, the one I voted for in KIP-1012 and what I understood to be the consensus in the 3.8.0 release thread. 1. If KIP-853 is feature-complete before October, Kafka 3.9 can be released ASAP with KIP-853. There will be no 3.10 release, an

Re: New release branch 3.9

My understanding was that the reason for the shorter cycle to the 3.9 release was based on the assumption that KIP-1012 would be ready soon, so we could get to 4.0 quicker. If we can't move to 4.0 sooner, what's to gain with an early 3.9? -- Igor

Re: [VOTE] KIP-1022 Formatting and Updating Features

I added this update to the end of the section Colin added. Justine On Tue, Jul 30, 2024 at 11:01 AM Jun Rao wrote: > Hi, Colin, > > Thanks for the update. We also excluded supported features with maxVersion > of 0 from both ApiVersionResponse and BrokerRegistrationRequest, and > excluded finali

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thanks Josep for your hard work! And to everyone who contributed to this release. Justine On Tue, Jul 30, 2024 at 8:04 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Thanks for running the release! > > On Tue, Jul 30, 2024 at 4:33 AM Colin McCabe wrote: > > > +1. Thanks, Jos

Re: New release branch 3.9

Hi Matthias, Note that is KIP-853 the one with the feature parity with ZK. KIP-1012 was the one were we agreed to stay on 3.x until feature parity. The reason to have a 3.10 is to have a safe way to upgrade to KRaft Kafkas while ZK is still around. We try to explain this in KIP-1012 --

[jira] [Resolved] (KAFKA-17203) StreamThread leaking producer instances

[ https://issues.apache.org/jira/browse/KAFKA-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17203. Fix Version/s: 3.10.0 Resolution: Fixed > StreamThread leaking producer instances >

Re: New release branch 3.9

Thanks for the input. However, I am wondering if releasing 3.9 makes sense if KIP-1012 won't make it? My understanding was we do 3.9 only to not delay 3.8 and not delay 4.0. If we would go with a 3.10, would it also be an "intermediate" release like 3.9? Or would it replace 4.0? For the first

Re: New release branch 3.9

One option is to just have a separate 4.0 branch for development to not block work on new features. With Apache Accumulo we have a similar situation where we have a separate branch (ironically also 4.0) that we are calling "elasticity" that is a long running br

[jira] [Resolved] (KAFKA-16972) Move `BrokerTopicStats` and `BrokerTopicMetrics` to `org.apache.kafka.storage.log.metrics` (storage module)

[ https://issues.apache.org/jira/browse/KAFKA-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16972. Fix Version/s: 3.10.0 Resolution: Fixed > Move `BrokerTopicStats` and `BrokerTopicM

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hello Rich, thanks for resurrecting the KIP, seems to fill a gap indeed. LM1. Specifically related to motivation#1. ProducerRecord already has a timestamp, passed into the RecordMetadata, that represents the creation time provided on new ProducerRecord, so couldn't we reuse it to avoid the extra c

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrick, I share Greg's concerns with the feature as it's currently proposed. I don't think I could vote for something unless it made replication of genuinely internal topics and replication cycles impossible, or at least significantly less likely. Best, Chris On Tue, Jul 30, 2024, 14:51 Gre

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrik, Thanks for the KIP! Your motivation for this KIP is reasonable, because it is definitely possible for the ".internal" suffix to collide with real topics. It would have been nice if the original design included some mm2-specific namespace like "mm2.internal" to lessen the likelihood of

[jira] [Resolved] (KAFKA-17185) Make sure a single logger instance is created

[ https://issues.apache.org/jira/browse/KAFKA-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17185. Fix Version/s: 3.10.0 Resolution: Fixed > Make sure a single logger instance is cre

[jira] [Created] (KAFKA-17221) Flaky test DedicatedMirrorIntegrationTest::testMultiNodeCluster

Chris Egerton created KAFKA-17221: - Summary: Flaky test DedicatedMirrorIntegrationTest::testMultiNodeCluster Key: KAFKA-17221 URL: https://issues.apache.org/jira/browse/KAFKA-17221 Project: Kafka

Re: [VOTE] KIP-1022 Formatting and Updating Features

Hi, Colin, Thanks for the update. We also excluded supported features with maxVersion of 0 from both ApiVersionResponse and BrokerRegistrationRequest, and excluded finalized features with version of 0 from ApiVersionResponse. It would be useful to document those too. Jun On Mon, Jul 29, 2024 at

[jira] [Resolved] (KAFKA-17175) Remove interface `BrokerNode` and `ControllerNode`

[ https://issues.apache.org/jira/browse/KAFKA-17175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17175. Fix Version/s: 3.9.0 Resolution: Fixed > Remove interface `BrokerNode` and `Control

Re: New release branch 3.9

+1 Greg. I'd be really happy to bump trunk to 4.0.0, but only once we know we can safely do so. On Tue, Jul 30, 2024 at 7:24 PM Greg Harris wrote: > Hi all, > > I agree that we are not yet ready for breaking changes on trunk, so I > opened a PR to bump to 3.10.0-SNAPSHOT: > https://github.com/ap

Re: New release branch 3.9

Hi all, I agree that we are not yet ready for breaking changes on trunk, so I opened a PR to bump to 3.10.0-SNAPSHOT: https://github.com/apache/kafka/pull/16732 When KIP-853 is feature complete, we can bump to 4.0.0-SNAPSHOT. Thanks, Greg On Tue, Jul 30, 2024 at 10:01 AM Josep Prat wrote: > H

[jira] [Resolved] (KAFKA-17044) Connector deletion can lead to resource leak during a long running connector startup

[ https://issues.apache.org/jira/browse/KAFKA-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-17044. --- Resolution: Won't Fix > Connector deletion can lead to resource leak during a long running c

Re: New release branch 3.9

Hi all, As per KIP-1012[1] we can't yet say if the next release will be 3.10.0 or 4.0.0. It will come down to the state of KIP-853 in 3.9.0. So, in my opinion we should still wait before committing breaking changes on trunk until we know for sure that KIP-853 will make it. Maybe Jose can share mor

Re: New release branch 3.9

Thanks for cutting the release branch. It's great to see `trunk` being bumped to 4.0-SNAPSHOT, and I wanted to follow up on this: We have a bunch of tickets that we can only ship with 4.0 release, and these tickets were blocked so far. I wanted to get confirmation that we will stick with 4.0

Re: New release branch 3.9

hi Colin, Could you please consider adding https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0 The issue is used to deprecate the formatters in core module. Also, it implements the replacements for them. In order to follow the deprecation rules, it would be nice to have KAFKA-1 in 3.9

Re: New release branch 3.9

Thanks Colin. For KIP-853 (KRaft Controller Membership Changes), we still have the following features that are in progress. 1. UpdateVoter RPC and request handling 2. Storage tool changes for KIP-853

[jira] [Created] (KAFKA-17220) Define new metrics for MirrorMaker2

Greg Harris created KAFKA-17220: --- Summary: Define new metrics for MirrorMaker2 Key: KAFKA-17220 URL: https://issues.apache.org/jira/browse/KAFKA-17220 Project: Kafka Issue Type: New Feature

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.8 #76

See Changes: -- [...truncated 523078 lines...] [2024-07-30T15:20:30.983Z] > Task :connect:json:publishToMavenLocal [2024-07-30T15:20:30.983Z] > Task :server:compileTestJava [20

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thanks for running the release! On Tue, Jul 30, 2024 at 4:33 AM Colin McCabe wrote: > +1. Thanks, Josep! > > Colin > > On Mon, Jul 29, 2024, at 10:32, Chris Egerton wrote: > > Thanks for running the release, Josep! > > > > > > On Mon, Jul 29, 2024, 13:31 'Josep Prat' via kafka-clients < > kafka-

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

Omnia, thanks for the updates! > Am happy to add section for throttling in this KIP if it is high concern or open a followup KIP for this once we already have the pagination in place. Which one do you suggest? I'm okay leaving throttling for a future KIP. It might be useful to see the feature in

Re: [PR] MINOR: Refresh of the docs [kafka-site]

mimaison merged PR #618: URL: https://github.com/apache/kafka-site/pull/618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

DA1: I like Jun's suggestion of using a wildcard, this would also help with the case I mentioned (cordon a whole broker, regardless of how many log dirs). DA2.1: Re: log dir names to UUID mapping -- do you mean a new CordonLogDirs RPC would need to send the UUID instead of the log dir name? DA2.2

Re: [PR] MINOR: Refresh of the docs [kafka-site]

jlprat commented on PR #618: URL: https://github.com/apache/kafka-site/pull/618#issuecomment-2258469770 This was the PR for trunk https://github.com/apache/kafka/pull/16654 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] MINOR: Refresh of the docs [kafka-site]

jlprat commented on PR #618: URL: https://github.com/apache/kafka-site/pull/618#issuecomment-2258463153 I thought I ported both there, maybe I overlooked a couple, I'll check later if nobody does -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] MINOR: Refresh of the docs [kafka-site]

mimaison commented on PR #618: URL: https://github.com/apache/kafka-site/pull/618#issuecomment-2258458423 Thanks for the review. So it looks like some changes were only done in kafka-site and are not in 3.8. I pushed an update to fix these. -- This is an automated message from the Apache

[jira] [Created] (KAFKA-17219) Adjust system test framework for new protocol consumer

Dongnuo Lyu created KAFKA-17219: --- Summary: Adjust system test framework for new protocol consumer Key: KAFKA-17219 URL: https://issues.apache.org/jira/browse/KAFKA-17219 Project: Kafka Issue Ty

Re: [PR] MINOR: Refresh of the docs [kafka-site]

jlprat commented on code in PR #618: URL: https://github.com/apache/kafka-site/pull/618#discussion_r1696998174 ## 38/upgrade.html: ## @@ -19,7 +19,7 @@

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3151

See

[PR] MINOR: Refresh of the docs [kafka-site]

mimaison opened a new pull request, #618: URL: https://github.com/apache/kafka-site/pull/618 Regenerated the docs from the 3.8 branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Team, I would like to start a discussion on KIP-1074: Make the replication of internal topics configurable The goal of this KIP is to make it easier to replicate topics that see

[jira] [Resolved] (KAFKA-15469) Document built-in configuration providers

[ https://issues.apache.org/jira/browse/KAFKA-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-15469. Fix Version/s: 4.0.0 Resolution: Fixed > Document built-in configuration providers

[jira] [Created] (KAFKA-17218) kafka-consumer-groups fails to describe all group if one group has been consuming from a deleted topic

Omnia Ibrahim created KAFKA-17218: - Summary: kafka-consumer-groups fails to describe all group if one group has been consuming from a deleted topic Key: KAFKA-17218 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [PR] Add KIP-477 to blog [kafka-site]

jlprat merged PR #617: URL: https://github.com/apache/kafka-site/pull/617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.o

Re: [PR] Add KIP-477 to blog [kafka-site]

jlprat commented on code in PR #617: URL: https://github.com/apache/kafka-site/pull/617#discussion_r1696798968 ## blog.html: ## @@ -110,6 +110,9 @@ Kafka Streams Kafka Connect +https://cwiki.apache.org/confluence/display/K

Re: [PR] Add KIP-477 to blog [kafka-site]

soarez commented on code in PR #617: URL: https://github.com/apache/kafka-site/pull/617#discussion_r1696796779 ## blog.html: ## @@ -110,6 +110,9 @@ Kafka Streams Kafka Connect +https://cwiki.apache.org/confluence/display/K

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #2

See

[jira] [Reopened] (KAFKA-15863) Handle push telemetry throttling with quota manager

[ https://issues.apache.org/jira/browse/KAFKA-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apoorv Mittal reopened KAFKA-15863: --- Reopening to introduce metric quota. > Handle push telemetry throttling with quota manager > --

[jira] [Resolved] (KAFKA-14614) Missing cluster tool script for Windows

[ https://issues.apache.org/jira/browse/KAFKA-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-14614. Fix Version/s: 3.6.0 Resolution: Fixed > Missing cluster tool script for Windows >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3150

See

[jira] [Resolved] (KAFKA-17210) Broker fixes for smooth concurrent fetches on share partition

[ https://issues.apache.org/jira/browse/KAFKA-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17210. --- Fix Version/s: 4.0.0 Resolution: Fixed > Broker fixes for smooth concurrent fetches o