Re: [VOTE] 3.9.1 RC1

Hi all, Update the system tests result link: https://drive.google.com/drive/folders/1lf7EH-m8PNPGfv7Fj3-FLnz_UH3un41n Summary: All the 1st run failed tests can pass after re-run. So they are just flaky. No issues found. Thanks. Luke On Wed, Apr 23, 2025 at 11:57 PM Kuan Po Tseng wrote: > Hi

[jira] [Created] (KAFKA-19193) Rack-aware partitioning for Kafka Producer

Ivan Yurchenko created KAFKA-19193: -- Summary: Rack-aware partitioning for Kafka Producer Key: KAFKA-19193 URL: https://issues.apache.org/jira/browse/KAFKA-19193 Project: Kafka Issue Type: Im

Re: [DISCUSS] KIP-1123: Rack-aware partitioning for Kafka Producer

Hi PoAn, Thank you, I created the Jira and updated the link in the KIP. Best, Ivan On Wed, Apr 23, 2025, at 10:57, PoAn Yang wrote: > Hi Ivan, > > Thanks for the KIP. The Jira link for this KIP is KAFKA-1. > Could you update to a correct Jira number? > > Best, > PoAn > > > On Mar 30, 2025, at

Re: [DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

Thank you for your reply The proposed load balancer is designed to distribute partitions uniformly across all brokers, assuming homogeneous hardware and similar partition workloads (i.e., CPU, network, and disk I/O demands are consistent). Under these conditions, it guarantees globally balanced pa

Re: [SPAM][DISCUSS] KIP-1164: Topic Based Batch Coordinator

Hi! Thanks for the greate work and I'm excited to see it happens. This KIP looks well to me. Seems Batch Coordinator is very important in the diskless implementation, could you explain more details on the implementation? I think it'll be much better to show what Batch Coordinator will do when w

[jira] [Created] (KAFKA-19192) Old bootstrap.metadata files cause problems with post-KAFKA-18601 servers

Colin McCabe created KAFKA-19192: Summary: Old bootstrap.metadata files cause problems with post-KAFKA-18601 servers Key: KAFKA-19192 URL: https://issues.apache.org/jira/browse/KAFKA-19192 Project: Ka

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

Hi Jun, > The issue described in the KIP applies to regular topics too. If the HWM is > delayed in the follower, it could delay the consumption for a client > fetching from a follower. It would be useful to document that too. Let me look at the code again, but I think that regular topics are not

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi, Jiunn-Yang, Thanks for the reply. Q2. It's true that group.coordinator.rebalance.protocols and process.roles configurations can't be empty right now. With this KIP, the user will probably get a different (and more accurate) exception. It will be useful to document that. Regarding cleanup.pol

Re: [DISCUSS] KIP-1166: Improve high-watermark replication

Hi, Jose, Thanks for the KIP. The issue described in the KIP applies to regular topics too. If the HWM is delayed in the follower, it could delay the consumption for a client fetching from a follower. It would be useful to document that too. Jun On Tue, Apr 22, 2025 at 12:30 PM José Armando Gar

[jira] [Created] (KAFKA-19191) Don't load bootstrap.checkpoint files if the cluster is already bootstrapped

José Armando García Sancio created KAFKA-19191: -- Summary: Don't load bootstrap.checkpoint files if the cluster is already bootstrapped Key: KAFKA-19191 URL: https://issues.apache.org/jira/browse/KAFKA

RE: [DISCUSS] KIP-1121: Compression acceleration in Kafka

Hi All, This is a gentle reminder about open discussion on this compression acceleration KIP. I would love to hear your thoughts. Thanks, Olasoji -Original Message- From: Denloye, Olasoji Sent: Monday, April 7, 2025 8:49 AM To: dev@kafka.apache.org Subject: RE: [DISCUSS] KIP-1121: Com

Re: [VOTE] 3.9.1 RC1

Hi TengYao, I did some testing on the upgrade path: - Set up a ZK-based Kafka cluster on 3.5, created some topics, and produced a bunch of messages - Upgraded to 3.9.1-RC1 - Migrated to KRaft And everything worked smoothly! Thanks a lot for all your hard work! Best, Kuan-Po Tseng On 2025/04/2

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

Hi Jun, Q2: For the group.coordinator.rebalance.protocols and process.roles configurations, setting them to an empty value is invalid—the broker/controller will fail to start in such cases. Therefore, I believe it's safe to assume that no users would configure them this way, and we can reasona

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

Hi Just coming back from my vacation. Sorry for the late reply. DJ03: Sure, will update the KIP for the naming change. DJ04: Good catch, will update. DJ05: Yeah, we can ignore this filter when it is null or an empty string. DJ06: Will mention it uses re2j KT01: Sure, will do. KT02: Will update. KT

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

hi Jiunn-Yang, Thanks for the KIP, and I understand that maintaining compatibility is important. However, according to the documentation, we don't allow an empty value for the cleanup.policy. Hence, should we consider throwing an exception for an empty value in version 4.x? This could streamline

[jira] [Resolved] (KAFKA-19118) Enable KIP-1071 in existing streams integration tests

[ https://issues.apache.org/jira/browse/KAFKA-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lucas Brutschy resolved KAFKA-19118. Resolution: Fixed > Enable KIP-1071 in existing streams integration tests > --

[jira] [Created] (KAFKA-19190) StreamsUncaughtExceptionHandlerIntegrationTest fails

Lucas Brutschy created KAFKA-19190: -- Summary: StreamsUncaughtExceptionHandlerIntegrationTest fails Key: KAFKA-19190 URL: https://issues.apache.org/jira/browse/KAFKA-19190 Project: Kafka Issu

Re:[DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

Thanks for this KIP (and the companion PR) This is an interesting proposal. I have to admit that I've only used this tool when I want to force a specific topic/ReplicaSet distribution. I wonder what criteria you used when deciding the set of goals and their order. Reason I ask is because we us

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

Hello Jun, At this stage, KAFKA-18390 only involves modifications to internal data structures. Interface-level changes are currently out of scope. If any interface changes in the future, they will be tracked under a separate JIRA. Best Regards, Jiunn-Yang > Mickael Maison 於 2025年4月23日 下午6:07

Re: [SPAM]Re: [DISCUSS] KIP-1150 Diskless Topics

Hi Yuxia! Thank you for the question. We've just opened the discussion thread for the KIP-1164 [1]. If you don't mind, could you please repost your question there? This would help a lot to keep the branchy discussion manageable. Best, Ivan [1] https://lists.apache.org/thread/m9l6lbqv2cffxtz5fr

Re: [VOTE] 3.9.1 RC1

Hi TengYao, I have done the following tests: - Built from 3.9.1-rc1 with OpenJDK-17 and OpenJDK-21. - Ran all unit and integration tests. - Verified basic produce/consume. - Validated the checksums and keys. - Ran KRaft with kafka_2.13-3.9.1.tgz Thanks, PoAn > On Apr 23, 2025, at 5:00 PM, TengY

Re: Re: [VOTE] KIP-1142: Allow to list non-existent group which has dynamic config

+1 (binding) Thanks, Chia-Ping TengYao Chi 於 2025年4月23日 週三 下午8:06寫道： > Thanks for the KIP. > +1 (binding) > > Best, > TengYao > > On 2025/04/14 11:29:28 Andrew Schofield wrote: > > Hi PoAn, > > Thanks for the KIP. > > > > +1 (binding) > > > > Andrew > >

RE: Re: [VOTE] KIP-1142: Allow to list non-existent group which has dynamic config

Thanks for the KIP. +1 (binding) Best, TengYao On 2025/04/14 11:29:28 Andrew Schofield wrote: > Hi PoAn, > Thanks for the KIP. > > +1 (binding) > > Andrew > > From: PoAn Yang > Sent: 08 April 2025 09:00 > To: dev@kafka.apache.org > Subject: [VOTE] KIP-11

Re: [DISCUSS] KIP-1150 Diskless Topics

Hi Christo and all! Thank you for your questions. > CL - 1: In the same lane as Luke's comment, it would be very useful to see > explicitly what will stay on disk and what won't stay on disk Good point. I added a small section "Disk usage and lack thereof" to KIP-1150. > CL - 2: It would also b

Re: [DISCUSS] KIP-1140: Avoid to return null value in Map from public api of consumer

Hi Kirk, Thanks for your comment. I agree that ‘allow.null.offsets.entries' is a better name. I've updated the KIP accordingly. Best Regards, Jiunn-Yang > Kirk True 於 2025年4月23日 上午8:20 寫道： > > Hi Jiunn-Yang, > > Thanks for the updates! > > IMO, "allow.null.entries" might be a little too va

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

Hi Jun, KIP-877 is not in a release yet. So assuming nobody objects to the change, we can do it before 4.1.0. Then any further changes will require another KIP. Thanks, Mickael On Tue, Apr 22, 2025 at 8:05 PM Jun Rao wrote: > > Hi, Jiunn-Yang, > > Thanks for the improvement. > > KAFKA-18390 ch

[jira] [Created] (KAFKA-19189) Handle deletion of __share_group_state topic in share coordinator

Andrew Schofield created KAFKA-19189: Summary: Handle deletion of __share_group_state topic in share coordinator Key: KAFKA-19189 URL: https://issues.apache.org/jira/browse/KAFKA-19189 Project: Ka

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

Bump On Mon, Apr 7, 2025 at 4:29 PM Mickael Maison wrote: > > Hi David, > > DA1: Done > > DA2.1: Yes because we only have the directory ids in the metadata log. > At the moment only brokers have the path -> id mappings. > > DA2.2: I proposed this mechanism because of DA2.1. Also this is > consist

[jira] [Resolved] (KAFKA-18629) DeleteShareGroups impl

[ https://issues.apache.org/jira/browse/KAFKA-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-18629. -- Fix Version/s: 4.1.0 Resolution: Fixed > DeleteShareGroups impl > -

[jira] [Resolved] (KAFKA-19156) Remove group.share.max.groups config

[ https://issues.apache.org/jira/browse/KAFKA-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-19156. -- Fix Version/s: 4.1.0 Resolution: Fixed > Remove group.share.max.groups config >

[jira] [Resolved] (KAFKA-18488) Additional protocol tests for share consumption

[ https://issues.apache.org/jira/browse/KAFKA-18488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-18488. -- Resolution: Fixed https://issues.apache.org/jira/browse/KAFKA-18794 tracks the resolut

[DISCUSS] KIP-1164: Topic Based Batch Coordinator

Hi all! We want to start the discussion thread for KIP-1164: Topic Based Batch Coordinator [1], which is a sub-KIP for KIP-1150 [2]. Let's use the main KIP-1150 discuss thread [3] for high-level questions, motivation, and general direction of the feature and this thread for discussing the batc

[DISCUSS] KIP-1163: Diskless Core

Hi all! We want to start the discussion thread for KIP-1163: Diskless Core [1], which is a sub-KIP for KIP-1150 [2]. Let's use the main KIP-1150 discuss thread [3] for high-level questions, motivation, and general direction of the feature and this thread for particular details of implementatio

Re: [DISCUSS] KIP-1150 Diskless Topics

Dear Community, I am truly delighted to see the KIP-1150 proposal put forth by Josep, which presents an exciting new architecture for the cloud economy. It was an honor to hear Ziming and Stanislav mention AutoMQ[1] during the discussion. Over the past few years, AutoMQ has dedicated itself to

[VOTE] 3.9.1 RC1

Hello Kafka users, developers, and client-developers, This is the second candidate for the release of Apache Kafka 3.9.1. This RC fixed the release tag off to a temporary branch issue (KAFKA-19166 ) compared with RC0. This is a bug-fix release wi

Re: [DISCUSS] KIP-1123: Rack-aware partitioning for Kafka Producer

Hi Ivan, Thanks for the KIP. The Jira link for this KIP is KAFKA-1. Could you update to a correct Jira number? Best, PoAn > On Mar 30, 2025, at 2:13 AM, Ivan Yurchenko wrote: > > Hi Federico, > > There's a sentence about reduced latency in the Motivation section. Do you > think that'd be eno

[jira] [Created] (KAFKA-19188) kafka-topics.sh --describe output for message.format.version is inconsistent immediately after cluster restart when not set at topic level

Fredrik Lysén created KAFKA-19188: - Summary: kafka-topics.sh --describe output for message.format.version is inconsistent immediately after cluster restart when not set at topic level Key: KAFKA-19188 URL: https:

Re: [DISCUSS] KIP-1119: Add support for SSL hot reload

Hi Moncef, Have you tried experimenting the behaviour of JDK's WatchService with bind mounts? Reading [0], it appears that inotify on linux doesn't emit events for bind mounts. This may pose a limitation in Kubernetes as that's how config maps are mounted. Another scenario to consider would b