Re: [DISCUSS] KIP-1112: allow custom processor wrapping

Thanks for the feedback all -- I have updated the KIP with everything mentioned so far, and added two static convenience methods for users to use to convert a ProcessorSupplier into a WrappedProcessorSupplier Let me know if you have any more questions or suggestions. By the way, since it was ment

[jira] [Created] (KAFKA-18054) Automatically detect missed configs needed by a topology

A. Sophie Blee-Goldman created KAFKA-18054: -- Summary: Automatically detect missed configs needed by a topology Key: KAFKA-18054 URL: https://issues.apache.org/jira/browse/KAFKA-18054 Project:

[jira] [Created] (KAFKA-18055) Remove TopologyConfig and clean up Topology/StreamsBuilder config APIs

A. Sophie Blee-Goldman created KAFKA-18055: -- Summary: Remove TopologyConfig and clean up Topology/StreamsBuilder config APIs Key: KAFKA-18055 URL: https://issues.apache.org/jira/browse/KAFKA-18055

[jira] [Created] (KAFKA-18053) Clean up TopologyConfig and API for supplying configs needed by the topology

A. Sophie Blee-Goldman created KAFKA-18053: -- Summary: Clean up TopologyConfig and API for supplying configs needed by the topology Key: KAFKA-18053 URL: https://issues.apache.org/jira/browse/KAFKA-18053

Re: [DISCUSS] Java 23 Support for 3.9.x

Dear all, It seems the main question is whether we should consider version 3.9 as an LTS release. For example, will we continue with versions like 3.9.1, 3.9.2, ... 3.9.100? If yes, we should backport both KIP-1006 and support for future JDKs. If not, backporting KIP-1006 to 3.9 would be suffici

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hi Lianet / Jeff, Thanks for review and suggestions. LM4: At group level, the protocol field is from ModernGroup#protocolType. It’s a string. The value is defined in different places like ShareGroup#PROTOCOL_TYPE and ConsumerProtocol#PROTOCOL_TYPE. In KIP-1043

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #124

See

Re: [DISCUSS] KIP-1111: Enforcing Explicit Naming for Kafka Streams Internal Topics

Thanks for the KIP. Overall this does sound useful. About Almog's comments (A1) -- I don't think that `topics.internal.require.explicit.naming` would be a good name, as we use `topic.` prefix for actual topic configs. Thus, KS would pickup `internal.require.explicit.naming` and try to apply i

[jira] [Resolved] (KAFKA-18019) Convert INVALID_PRODUCER_ID_MAPPING from abortable error to fatal error

[ https://issues.apache.org/jira/browse/KAFKA-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim resolved KAFKA-18019. -- Fix Version/s: 4.0.0 Resolution: Fixed > Convert INVALID_PRODUCER_ID_MAPPING from abortable err

Re: [DISCUSS] PR workflow

Nice! On 11/20/24 10:09 AM, David Arthur wrote: Hey folks, I have raised a PR that adds the notion of "triage" to our Pull Requests. https://github.com/apache/kafka/pull/17881/files The tldr is that new PRs from non-committers will get labeled with "triage" and then with "needs-attention" aft

Re: [DISCUSS] KIP-1112: allow custom processor wrapping

Bruno -- I updated the javadocs to remove the wrapping requirement. Thank you for pointing this out Almog -- yeah that makes a lot of sense to me, especially being able to add methods for the wrapper without having to add anything to the ProcessorSupplier classes. I'm updated the KIP with this sug

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi, Divij, Thanks for the updated KIP. Regarding num.recovery.threads.per.data.dir/num.network.threads/num.io.threads, I have a few concerns. 1. It seems that we can't use floor since this can result in a value of 0, which is invalid. 2. I am not sure that the new default value is actually better

Re: [DISCUSS] KIP-1112: allow custom processor wrapping

Hi Sophie, I was thinking it might also be good for us to return from the wrapped methods new marker interfaces like WrappedProcesserSupplier and WrappedFixedKeyProcesserSupplier. These will implement ProcesserSupplier and FixedKeyProcesserSupplier respectively with no additional methods (yet), bu

Re: [ANNOUNCE] Apache Kafka 3.9.0

Bruno, Thanks for uploading the artifacts to s3 as we discussed offline :) Divij, I have edited 3.9.0 in JIRA so that it now shows up as "released." Also, 3.6.2 has been archived, in accordance with our usual policy. The website points to our archive links already. Chia-Ping Tsai, yes, all subs

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hi PoAn, Thanks for the KIP. I have some questions: JK1: (Realized this is similar to LM5) On `--describe --verbose` proposed changes - doesn't `--describe $group` default to printing the offsets? Perhaps you're referring to the `--state` argument? Also, would that mean the default `--describe

Re: [VOTE] KIP-1081: Graduation Steps for KIPs

Hi Josep, If the level numbers are not to be used, then let's remove them from the KIP. (Also, I guess this is water under the bridge now, but I disagree that they were ever useful in the discussion; to me, they just added confusion) I also agree with Ismael that API stability should be up to

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hello PoAn, just of couple more minor comments: LM4. Regarding the new “protocol” field added to MemberDescription. Should we consider reusing the existing GroupProtocol enum instead of String? (It’s the one we use from the consumer side to refer to the protocol in use, just missing Share I notice

[jira] [Resolved] (KAFKA-18044) Test ticket for INFRA

[ https://issues.apache.org/jira/browse/KAFKA-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-18044. Resolution: Not A Problem > Test ticket for INFRA > -

Re: [VOTE] KIP-1112: allow custom processor wrapping

Hi, +1 Thanks, Bruno On 19.11.24 22:45, Guozhang Wang wrote: Thanks Sophie, +1 on this. Would be a useful wrapper. Guozhang On Tue, Nov 19, 2024 at 3:06 AM Lucas Brutschy wrote: Thanks! +1 (binding) Lucas Bill Bejeck schrieb am Di., 19. Nov. 2024, 00:06: Thanks for the KIP, Sophie,

Re: [DISCUSS] KIP-1112: allow custom processor wrapping

Hi Sophie, Thanks for the details! Yeah, I think replacing the contract in the javadocs with a warning is better. Best, Bruno On 20.11.24 10:05, Sophie Blee-Goldman wrote: Have you considered to add an interface that instead of wrapping a processor adds methods for pre-processing and post-

[jira] [Created] (KAFKA-18052) Decouple the dependency of feature stable version to the metadata version

Dongnuo Lyu created KAFKA-18052: --- Summary: Decouple the dependency of feature stable version to the metadata version Key: KAFKA-18052 URL: https://issues.apache.org/jira/browse/KAFKA-18052 Project: Kafk

Re: [DISCUSS] Java 23 Support for 3.9.x

> Has the SecurityManager been fully removed in JDK 23? > What is the effect of running Kafka 3.9.0 with JDK 23? The SecurityManager has been degraded, so by default our users experience an UnsupportedOperationException. They can work-around this by setting a system property. In JRE 24, JEP-486 [1

Re: [DISCUSS] Java 23 Support for 3.9.x

Greg, I have not been following this closely, so apologies for some basic questions. Has the SecurityManager been fully removed in JDK 23? What is the effect of running Kafka 3.9.0 with JDK 23? By "4.0 breaking changes" do you mean changes to our JDK/Scala supported versions, removal or ZK, Kaf

[jira] [Created] (KAFKA-18051) Disallow creating ACLs with principals that do not contain a colon

Colin McCabe created KAFKA-18051: Summary: Disallow creating ACLs with principals that do not contain a colon Key: KAFKA-18051 URL: https://issues.apache.org/jira/browse/KAFKA-18051 Project: Kafka

[DISCUSS] PR workflow

Hey folks, I have raised a PR that adds the notion of "triage" to our Pull Requests. https://github.com/apache/kafka/pull/17881/files The tldr is that new PRs from non-committers will get labeled with "triage" and then with "needs-attention" after 7 days. The purpose is to raise visibility of no

Re: [DISCUSS] KIP-1109: Unifying Kafka Consumer Topic Metrics

Hi, Chia-Ping, We could deprecate those metrics too. Then, we probably want to do a formal KIP on this. Thanks, Jun On Tue, Nov 19, 2024 at 3:11 AM Chia-Ping Tsai wrote: > hi Apoorv, > > Should we follow the deprecation cycle in this KIP? I mean, we should keep > both metrics for now and add

RE: [DISCUSS] Java 23 Support for 3.9.x

We (Trino project) started testing JDK 24 compatibility recently and got blocked for testing Kafka connector due to a Security Manager removal in JDK 24 build 22 for which there is no workaround. We'd love to continue supporting the Kafka connector, but forcing users to upgrade to 4.0+ compatible

[DISCUSS] KIP-1113: Enhancements to Rack-Aware Partition Assignor for Improved Client-Side Independence and Load Balancing

Hello, Everyone! Hope you are doing well! This is new KIP-1113 under discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1113%3A+Enhancements+to+Rack-Aware+Partition+Assignor+for+Improved+Client-Side+Independence+and+Load+Balancing This proposal will provide another option for rac

[jira] [Resolved] (KAFKA-18027) MINOR: Correct DelayedOperationPurgatory code around adding of an already completed operation

[ https://issues.apache.org/jira/browse/KAFKA-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-18027. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk. > MINOR: Correct DelayedOperation

[DISCUSS] Java 23 Support for 3.9.x

Hi all, Now that 3.9.0 is released and 4.0.x is progressing, I'd like to understand everyone's expectations about the 3.9.x branch, and ask for a specific consensus on Java 23 support. Some context that I think is relevant to the discussion: * KIP-1006 [1] proposes a backwards-compatible strategy

Re: [PR] MINOR: Fix the leak "unknown" group.coordinator.rebalance.protocols on document [kafka-site]

m1a2st commented on PR #648: URL: https://github.com/apache/kafka-site/pull/648#issuecomment-2486215273 This PR is associate with [PR](https://github.com/apache/kafka/pull/17834). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi Divij, We would like to change the default value for the below remote log configurations: 1. remote.log.manager.copier.thread.pool.size -- default value from -1 to 10 2. remote.log.manager.expiration.thread.pool.size -- default value from -1 to 10 In KIP-950

[jira] [Resolved] (KAFKA-17721) Enable to configure listener name and protocol for controller

[ https://issues.apache.org/jira/browse/KAFKA-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17721. Fix Version/s: 4.0.0 Resolution: Fixed > Enable to configure listener name and prot

[jira] [Created] (KAFKA-18050) Upgrade the checkstyle version to 10.20.1

黃竣陽 created KAFKA-18050: --- Summary: Upgrade the checkstyle version to 10.20.1 Key: KAFKA-18050 URL: https://issues.apache.org/jira/browse/KAFKA-18050 Project: Kafka Issue Type: Improvement R

Re: [DISCUSS] Apache Kafka 4.0.0 release

Hi Divij, I agree we _should_ include it into v4.0.0 to make the kafka default value more reasonable, instead of doing it until v5.0.0. What do you think, @David Jacot ? Thanks. Luke On Tue, Nov 19, 2024 at 7:01 PM Divij Vaidya wrote: > Hey David > > Can I please request an exceptional extens

[jira] [Created] (KAFKA-18049) Update the caffeine version to 3.1.8

黃竣陽 created KAFKA-18049: --- Summary: Update the caffeine version to 3.1.8 Key: KAFKA-18049 URL: https://issues.apache.org/jira/browse/KAFKA-18049 Project: Kafka Issue Type: Improvement Repor

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #111

See

Re: [DISCUSS] Apache Kafka 4.0.0 release

Based on initial feedback from the community, I don't anticipate the discussion for KIP-1030 to go beyond 2 weeks. Will keep you posted. -- Divij Vaidya On Wed, Nov 20, 2024 at 11:03 AM David Jacot wrote: > Hi Divij, > > I think that we can make a small exception for KIP-1030 but we cannot wa

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Thank you everyone for your feedback. The new revision of KIP is ready for review. Notable changes since last revision: 1. Removed change to log.message.timestamp.before.max.ms 2. Modified default for num.recovery.threads.per.data.dir 3. Added new proposals for num.io.threads and num.network.threa

[jira] [Created] (KAFKA-18047) Add org.apache.kafka.automatic.config.providers to System Properties section

Mickael Maison created KAFKA-18047: -- Summary: Add org.apache.kafka.automatic.config.providers to System Properties section Key: KAFKA-18047 URL: https://issues.apache.org/jira/browse/KAFKA-18047 Proj

Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker

Hi everyone, Bumping this - I think this gap affects the lives of many MM2 users, would be great to fill it. Any comments are welcome. TIA Daniel Dániel Urbán ezt írta (időpont: 2024. nov. 15., P, 13:05): > Hi Vidor, > > KV2 and KV3. I think it is a common setup, but it is usually described as >

[jira] [Created] (KAFKA-18048) Consider to decouple AppEventHandler from ConsumerNetworkThread

PoAn Yang created KAFKA-18048: - Summary: Consider to decouple AppEventHandler from ConsumerNetworkThread Key: KAFKA-18048 URL: https://issues.apache.org/jira/browse/KAFKA-18048 Project: Kafka Is

Re: [DISCUSS] Apache Kafka 4.0.0 release

Hi all, This is a friendly reminder that the KIP freeze deadline is today. As we have a few KIPs getting close to be accepted, I will wait until Monday to compile the final list. Please update the status of your KIPs. The next deadline is the Features freeze on Dec 11th. Cheers, David Cheers, D

Re: [DISCUSS] Apache Kafka 4.0.0 release

Hi Divij, I think that we can make a small exception for KIP-1030 but we cannot wait too long. How long do you need to close it? Ideally, we should aim for closing it in the next two weeks. If we cannot do it, we may have to discuss it again. Please keep me posted. BR, David On Wed, Nov 20, 2024

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi, This proposal looks good to me with new defaults and constraints. About breaking backward compatibility I've question on how you see it best handled in case someone upgrades Kafka cluster from 3.9 to 4.0 with let's say one topic having segment.bytes=500 kb (new min is 1 MB). Does it mean that

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

Hi Divij, One comment: For the default value of "num.recovery.threads.per.data.dir", we change from 1 to 2. Could we change the value based on the CPU core number? Ex: default value = CPU core num / num of log dirs. WDYT? Luke On Wed, Nov 20, 2024 at 5:53 PM Tommi Vainikainen wrote: > Hi, >

[jira] [Created] (KAFKA-18046) High CPU usage when using Log4j2

Anthony Maire created KAFKA-18046: - Summary: High CPU usage when using Log4j2 Key: KAFKA-18046 URL: https://issues.apache.org/jira/browse/KAFKA-18046 Project: Kafka Issue Type: Improvement

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

hi TaiJuWu Is there a possibility to extend this KIP to include topic-level compression for the producer? This is another issue that prevents us from sharing producers across different threads, as it's common to use different compression types for different topics (data). Best, Chia-Ping On 2

[jira] [Created] (KAFKA-18045) Add 0.11, 1.0, 1.1, and 2.0 back to streams_upgrade_test.py

Chia-Ping Tsai created KAFKA-18045: -- Summary: Add 0.11, 1.0, 1.1, and 2.0 back to streams_upgrade_test.py Key: KAFKA-18045 URL: https://issues.apache.org/jira/browse/KAFKA-18045 Project: Kafka

[jira] [Created] (KAFKA-18044) Test ticket for INFRA

Gavin McDonald created KAFKA-18044: -- Summary: Test ticket for INFRA Key: KAFKA-18044 URL: https://issues.apache.org/jira/browse/KAFKA-18044 Project: Kafka Issue Type: Wish Componen

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Thanks Lucas! That all sounds good to me. I think I officially have nothing left to say or ask about this KIP, so once you've updated the doc with what we've discussed in the past few messages then I'm personally feeling ready to vote on it. On Tue, Nov 19, 2024 at 11:10 PM Lucas Brutschy wrote:

Re: [DISCUSS] KIP-1112: allow custom processor wrapping

> > Have you considered to add an interface that instead of wrapping a > processor adds methods for pre-processing and post-processing to the > processor? The problem is that we actually want to be able to access all parts of the ProcessorSupplier. This includes things like inspecting the StoreBu

Re: [PR] MINOR: Fix the leak "unknown" group.coordinator.rebalance.protocols on document [kafka-site]

chia7712 merged PR #648: URL: https://github.com/apache/kafka-site/pull/648 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache