Re: Handling Private Methods in AclAuthorizer and Transition to StandardAuthorizer

Hi Kasper, > Starting with Kafka v3.8.0, methods like logAuditMessage and isSuperUser in AclAuthorizer were made private, which breaks our implementation. > 1. Is this change intentional, and if so, is there a recommended way to handle logging or superuser checks in custom authorizers? I checked t

Re: [VOTE] 3.7.2 RC1

Hi Matthias, Thanks for running the release. I did the following steps: - Validated the checksums and keys - Browsed the javadocs - Browsed the documentation - Ran all the quickstart using KRaft with kafka_2.13-3.7.2.tgz - Ran basic quickstart using 3.7.2-rc1 docker image +1 bind

Re: [ANNOUNCE] New committer: Andrew Schofield

Congratulations! Well deserved! Luke On Tue, Dec 3, 2024 at 5:55 AM Matthias J. Sax wrote: > Congrats! > > On 12/2/24 12:02 PM, Kirk True wrote: > > Congratulations Andrew Well-deserved. > > > > On Mon, Dec 2, 2024, at 6:13 AM, David Arthur wrote: > >> The Project Management Committee (PMC)

Re: Kafka "kafka-metadata-quorum.sh" regression in 3.9.0

Hi Jesús, That's part of the change in KIP-853: https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes#KIP853:KRaftControllerMembershipChanges-describe--status Thanks. Luke On Tue, Nov 12, 2024 at 11:48 PM Jesus Cea wrote: > In Kafka 3.8.1 I see this:

Re: [ANNOUNCE] Apache Kafka 3.9.0

> Hsu, José Armando García Sancio, Josep Prat, Jun Rao, Justine Olshan, Kamal > Chandraprakash, Ken Huang, Kevin Wu, Kirk True, Kondrat Bertalan, Krishna > Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Kuan-Po Tseng, Lee Dongjin, > Lianet Magrans , Logan Zhu, Loïc GREFFIER, Lucas Brutsc

Re: Kafka Tiered Storage Disablement Implement in ZK Mode Question

if just update the log configs > without a v5 stopReplicas v5 rpc from controller to broker in zk mode? And > why zk mode is not supported to disable? > Really appreciate if Luke Chen could help to answer, > thanks in advance! > > > Jianfeng Lin from Shopee >

Re: Inquiry on Scale Down Process for KRaft Kafka Cluster

Hi hayoung, This is the problem that KIP-1066 is trying to address. It is under discussion, and should be included in the upcoming releases. Thanks. Luke On Tue, Oct 29, 2024 at 12:4

Re: [DISCUSS] bug fix to allow dynamically change some log configs

figurations. I will fix the problem where static synonyms are > ignored sometimes, though, since that seems more "surprising" and bad. > > Maybe we should have removed those synonyms in 4.0, but I guess that ship > has sailed. Perhaps 5.0 :) > > best, > Colin > >

[DISCUSS] bug fix to allow dynamically change some log configs

Hi all, KAFKA-17584 is about a bug related to the incorrect synonym handling for dynamic log configurations. While there is already a PR to fix the issue, we found there are some log configs that shoul

Re: [VOTE] 3.9.0 RC0

Same here, no 3.9.0 artifacts in the staging repo. Thanks. Luke On Wed, Sep 18, 2024 at 4:49 PM Jakub Scholz wrote: > Hi Colin, > > Thanks for the RC. > > It looks like the Maven artifacts are missing for 3.9.0 in the staging > repo. Are we still waiting for some staging job to complete? Or is

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

ke On Thu, Aug 29, 2024 at 10:39 AM Dima Brodsky wrote: > Thanks Luke, > > I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. > > ttyl > Dima > > > On Wed, Aug 28, 2024 at 7:36 PM Luke Chen wrote: > > > Hi Dima, > > > > > I assume that

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Aug 29, 2024 at 1:58 AM Dima Brodsky wrote: > Hi Luke, > > I assume that unclean leader election exists as a static config in kafka > kraft, but is still unavailable dynamically? Or is it unavailable in > general? > > Thanks! > ttyl > Dima > > > On Wed, Aug 2

[DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Hi all, In v3.9.0, we're planning to support the `unclean.leader.election.enable` config in KRaft. Currently, the implementation PR is under review now. But I'd like to raise the discussion to let you know there will be one behavior change when enabling

Re: Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

Hi Dethmix, > The interesting thing is that this doesn't happen if I rollback inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is able to start and read all the data from the __consumer_offsets partitions with metadata version 3. The question is that expected behavior? I t

Re: KRaft Migration Prod Ready?

Hi Upesh, Yes, it's supported. The doc will be updated soon (after v3.8.0 released). Thanks. Luke On Wed, Jul 3, 2024 at 6:02 AM Upesh Desai wrote: > Hello, > > > > We have been a bit confused as to if KRaft migration from ZooKeeper is > production ready? It seems that the Kafka docs still ind

Re: AW: How do I resolve an UnwritableMetadataException

Hi all, Thanks for reporting this issue. This issue is fixed in KAFKA-16583 and will be included in v3.7.1 and v3.8.0, which should be released in June or July. Thanks. Luke On Tue, Jun 4, 2024 at 11:29 PM Sejal Patel wrote: > Thank you, Tobi

Re: Kafka 3.7 Documentation

t; Edgar Zubel > DevOps Engineer > > > -Original Message- > From: Luke Chen > Sent: Tuesday, May 28, 2024 9:48 AM > To: users@kafka.apache.org > Subject: Re: Kafka 3.7 Documentation > > Hi Edgar, > > Thanks for reporting this. > Yes, I confirmed the document i

Re: Kafka 3.7 Documentation

Hi Edgar, Thanks for reporting this. Yes, I confirmed the document is not correct. What we want is to revert from "Migrating brokers to KRaft", to "Enter Migration Mode on the Brokers" state, and in the end, Pure ZK. About the authorizer change, I'm fine if we want to mention it. I've created KAF

Re: Kafka retention bug (?)

Hi Nicholas, I didn't know anything in v3.7.0 would cause this issue. It would be good if you could open a JIRA for it. Some info to be provided: 1. You said "in the past", what version of Kafka was it using? 2. What is your broker configuration? 3. KRaft mode? Combined mode? (controller + broker

Re: [ANNOUNCE] New committer: Igor Soarez

Congrats, Igor! On Thu, Apr 25, 2024 at 6:10 AM Matthias J. Sax wrote: > Congrats! > > On 4/24/24 2:29 PM, Bill Bejeck wrote: > > Congrats Igor! > > > > -Bill > > > > On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley wrote: > > > >> Congratulations Igor! > >> > >> On Thu, 25 Apr 2024 at 6:27 AM, Chia

Re: Kraft controller readiness checks

ith the URP readiness check > and the PDB, we are guaranteed not to cause any problem read or write > errors. Rotations are rather long, but we don’t really care about speed. > > Thanks, > > Frank > > -- > Francesco Burato | Software Development Engineer | Adobe | > bur

Re: Kraft controller readiness checks

Hello Frank, That's a good question. I think we all know there is no "correct" answer for this question. But I can share with you what our team did for it. Readiness: controller is listening on the controller.listener.names The rationale behind it is: 1. The last step for the controller node sta

Re: production ready for zookeeper to kraft migration

Hi Matthieu, Yes, the ZK migrating to KRaft feature is already GA in v3.6.0. Sorry, we forgot to update the document in the Kafka-site repo. I've filed a PR for it: https://github.com/apache/kafka-site/pull/594 Thanks. Luke On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou wrote: > I looked at the

Re: Replicas not equally distributed within rack

Hi Abhishek, For Zookeeper's mode, there's no workaround, unfortunately. But you can upgrade your cluster to the latest kafka version (v3.7.0) to migrate to KRaft mode. For your information, in KRaft mode, we use `StripedReplicaPlacer`, which will fix your problem. ref: https://github.com/apache/k

Re: Possible bug on Kafka documentation

Hi Federico, Thanks for reporting the issue. We've fixed that in v3.5 and later in this PR: https://github.com/apache/kafka/pull/13115. But we didn't update for the older versions of docs. Are you willing to file a PR to kafka-site repo to fix that? Or create a JIRA issue for it? Thanks. Luke O

Re: Can a controller in a kafka kraft cluster be a bootstrap server

node is not routing to other running > node, It takes me always to restart the services, > Running on kafka version 3.2.1 (kraft mode) > > On Mon, Dec 11, 2023 at 12:33 PM Luke Chen wrote: > > > Hi Dima, > > > > You can set "process.roles=controller,broker" t

[ANNOUNCE] Apache Kafka 3.5.2

, Levani Kokhreidze, Lucas Brutschy, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya We welcome your help and feedback. For more information on how to

Re: Kafka frequent disconnection issue

Hi Ankit, We can't see the log snippet. But it looks normal to disconnect when connections.max.idle.ms expires. When increasing the connections.max.idle.ms value, there might be some activities in the connection during this time (ex: at 5 min 10 sec), so the idle timer is reset. Thanks. Luke On

Re: Can a controller in a kafka kraft cluster be a bootstrap server

Hi Dima, You can set "process.roles=controller,broker" to get what you want. Otherwise, the controller role cannot be served as a broker. Thanks. Luke On Sat, Dec 9, 2023 at 3:59 AM Dima Brodsky wrote: > Hello, > > Would the following configuration be valid in a kafka kraft cluster > > So lets

Re: [ANNOUNCE] Apache Kafka 3.6.1

a, > Federico Valeri, Greg Harris, Guozhang Wang, Hao Li, hudeqi, > iit2009060, Ismael Juma, Jorge Esteban Quilcate Otoya, Josep Prat, > Jotaniya Jeel, Justine Olshan, Kamal Chandraprakash, kumarpritam863, > Levani Kokhreidze, Lucas Brutschy, Luke Chen, Manikumar Reddy, > Matthias J. Sax,

Re: Kafka 2.7.2 to 3.5.1 upgrade

Hi Lud, This is a known issue(KAFKA-15353 ) and I've fixed it in v3.5.2 (will get released soon) and v3.6.0. Thanks. Luke On Mon, Dec 4, 2023 at 6:01 PM Lud Antonie wrote: > Hi Megh, > > No, the number of partitions haven't increased. > The up

[VOTE] 3.5.2 RC1

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.5.2. This is a bugfix release with several fixes since the release of 3.5.1, including dependency version bumps for CVEs. Release notes for the 3.5.2 release: https://home.apache.org/~s

Re: How to dynamically change configurations in the controllers

Hi Jesus, KIP-919 is what you're looking for: https://cwiki.apache.org/confluence/display/KAFKA/KIP-919%3A+Allow+AdminClient+to+Talk+Directly+with+the+KRaft+Controller+Quorum+and+add+Controller+Registration This feature will be included in next release (i.e. Kafka v3.7.0). Thanks. Luke On Thu,

Re: [ANNOUNCE] Apache Kafka 3.6.0

gmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang > > Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin, > > iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson, > > Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Es

Re: [ANNOUNCE] Apache Kafka 3.6.0

Gustafson, > Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate > Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio, > Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True, > Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstre

Re: Kafka Protocol : Compact Array or Array ?

Hi Neeraj, Yes, for MetadataRequest, version 0 ~ 8, the topic is ARRAY type. After version 9, it'll be COMPACT_ARRAY. It's because of this definition: "flexibleVersions": "9+". You can check KIP-482 for more information: https://cwiki.apache.org/confluence/display/KAFKA/KIP-482 Thanks. Luke On M

Re: [VOTE] 3.6.0 RC2

Hi Satish, I verified with: 1. Ran quick start in KRaft for scala 2.12 artifact 2. Making sure the checksum are correct 3. Browsing release notes, documents, javadocs, protocols. 4. Verified the tiered storage feature works well. +1 (binding). Thanks. Luke On Mon, Oct 2, 2023 at 5:23 AM Jakub

Re: [kafka-clients] [VOTE] 3.6.0 RC1

Hi Jose, Sounds good to me. Let's have further discussion in JIRA/PR, and target to v3.6.1/v3.7.0. Thanks. Luke On Tue, Sep 26, 2023 at 1:35 AM José Armando García Sancio wrote: > On Sat, Sep 23, 2023 at 3:08 AM Luke Chen wrote: > > > > Hi Satish, > > >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

; critical paths? > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489 > > > > Thanks, > > Satish. > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen wrote: > > > > > > Hi Satish, > > > > > > I found the current

Re: [kafka-clients] [VOTE] 3.6.0 RC1

Hi Satish, I found the current KRaft implementation will have "split brain" issue when network partition happens, which will cause inconsistent metadata returned from the controller. Filed KAFKA-15489 for this issue, and PR

Re: Ruquest for Jira assign permissions

Hi Romulo, You should be able to assign tasks to yourselves. Thanks. Luke On Sat, Aug 5, 2023 at 9:10 PM Romulo Oliveira wrote: > I added a new Jira task in order to contribute to a new feature for Kafka > Connectors TimestampConverter transformer: > https://issues.apache.org/jira/browse/KAFKA

Re: Unable to launch KRaft

Hi Adrian, Please follow the quick start guide in Kafka doc: https://kafka.apache.org/quickstart Thanks. Luke On Fri, Jul 28, 2023 at 3:53 PM adrien ruffie wrote: > dear all, > > I would like to try KRaft in but I didn't arrived to finish the > installation bellow: > > > https://developer.con

Re: [VOTE] 3.5.1 RC1

Hi Divij, I've run: 1. Download kafka_2.12-3.5.1.tgz 2. Run quick start using KRaft mode 3. Verified the checksum 4. Sanity check the javadoc All looks good. +1 (binding) Thanks. Luke On Tue, Jul 18, 2023 at 5:15 AM Chris Egerton wrote: > Hi Divij, > > Thanks for running this release! > > To

Re: Replication factor for a topic Increase

gards, > GP > > On Thu, 13 Jul, 2023, 13:54 Luke Chen, wrote: > > > Hi Gaurav, > > > > > Why do I see topic ReplicationFactor:1 during description of a topic ? > > I think you should set this config: `default.replication.factor` > > > > > And how

Re: Replication factor for a topic Increase

Hi Gaurav, > Why do I see topic ReplicationFactor:1 during description of a topic ? I think you should set this config: `default.replication.factor` > And how can we increase replication factor? I have 3 Kafka brokers running on 2.7.0 version. Please check this doc: https://kafka.apache.org/docu

Re: [KRaft] Clarification about Deploying Considerations (Kafka docs)

Hi Grigorios I've asked the author in the JIRA: https://issues.apache.org/jira/browse/KAFKA-14207 Thanks. Luke On Wed, Jul 5, 2023 at 10:31 PM Grigorios Avgitidis wrote: > Hi, > > On the KRaft section of the Kafka documentation, related with `Deploying > Considerations`, in one of the points i

Re: [ANNOUNCE] Apache Kafka 3.5.0

a, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff > Kim, Jim Galasyn, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando > García Sancio, Juan José Ramos, Junyang Liu, Justine Olshan, Kamal > Chandraprakash, Kirk True, Kowshik Prakasam, littlehorse-eng, liuzc9, Lucas > Brutschy, Luci

Re: Process to Upgrade Zookeeper from 2.7.0 to 3.4.1

Hi Gaurav, Please check Zookeeper's doc for upgrading guide. Thanks. Luke On Wed, Jun 14, 2023 at 12:03 PM Gaurav Pande wrote: > Hi Guys, > > Could anyone help on this query? > > Regards, > Gaurav > > On Tue, 13 Jun, 2023, 11:40 Gaurav Pande, wrote: > > > Hello Guys, > > > > Iam new in this s

[ANNOUNCE] Apache Kafka 3.4.1

, José Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash Mayya We welcome your help and feedback. For more information on how to report problems

Re: kafka 3.4.0 sasl_PLAINTEXT kafka-metadata-quorum.sh Unexpected Kafka request of type METADATA during SASL handshake

Hi, I've replied you in your previous email. https://lists.apache.org/thread/ogzpxtnsyklp7q82xh5t0y58rgtpc18x Please take a look. Thanks. Luke On Tue, Jun 6, 2023 at 5:13 PM Yj Jia wrote: > > > kafka 3.4.0 sasl_PLAINTEXT exec kafka-metadata-q

Re: [VOTE] 3.5.0 RC1

Hi Mickael, I ran the following validation steps: - Built from source with Java 17 and Scala 2.13 - Signatures and hashes of the artifacts generated - Navigated through Javadoc including links to JDK classes - Run the quickstart in KRaft and Zookeeper mode +1 (binding) from me. Thanks for running

Re: [VOTE] 3.4.1 RC3

since I couldn't find an existing issue for this one. I note that > others > > seem to have run the integration tests without problems, so I don't think > > this is a blocker. I also did the Kafka, Connect and Streams quickstarts. > > > > +1 binding. > > > &

Re: kafka 3.4.0 sasl_PLAINTEXT exec kafka-metadata-quorum.sh Unexpected Kafka request of type METADATA during SASL handshake.

Hi Yj, Thanks for reporting the issue. This is a known issue will be fixed in v3.4.1, which is planning to release next week (hopefully). JIRA: https://issues.apache.org/jira/browse/KAFKA-14711 Thanks. Luke On Thu, Jun 1, 2023 at 9:02 PM Yj Jia wrote: > kafka 3.4.0 sasl_PLAINTEXT exec kafka-me

Re: [VOTE] 3.4.1 RC3

> > On Sat, May 27, 2023 at 12:58 PM Jakub Scholz wrote: > > > > > > +1 (non-binding) ... I used the staged binaries and Maven artifacts to > > run > > > my tests and all seems to work fine. > > > > > > Thanks for running the release. >

Re: client.dns.lookup=”use_all_dns_ips” cache ttl

Hi George, I believe the JVM will have cache to this DNS records. And maybe OS will also cache it. Had a quick look, Kafka doesn't cache it. Thanks. Luke On Thu, Jun 1, 2023 at 1:21 AM George Goh wrote: > Hi! > > I’m considering to use the following configuration for my producers and > consume

[VOTE] 3.4.1 RC3

Hello Kafka users, developers and client-developers, This is the 4th candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 Proce

[VOTE] 3.4.1 RC2

Hello Kafka users, developers and client-developers, This is the 3rd candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 Proce

Re: Re: Re: Kafka 3.2.1 performance issue with JDK 11

Hi Vic, Yes, JDK 8 is still supported in kafka 3.x.x. It'll become unsupported after kafka 4.0.0. Thanks. Luke On Tue, May 23, 2023 at 9:56 AM Vic Xu wrote: > Hi Greg > > I know JDK 8 will solve the problem certainly, but I wondering if JDK8 has > been deprecated after Kafka 3? Can I use JDK 8

[VOTE] 3.4.1 RC1

Hello Kafka users, developers and client-developers, This is the 2nd candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 Proce

Re: [VOTE] 3.4.1 RC0

. > >>> > >>> Looks like the Maven artifacts are not in staging: > >>> > >>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/ > >>> > >>> Documentation still has 3.4.0, instead of 3.4.1 (no

Re: kafka acl issue

Hi Hari, You might want to ask in the client repo (kafkajs?) They should be able to help you. Thanks. Luke On Fri, May 19, 2023 at 3:00 PM HariBabu kuruva wrote: > Hi All, > > I am trying to implement kafka acl for one of the topics. > it's a kafka cluster with 1 broker. > > Below are the ACL'

[VOTE] 3.4.1 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 Pro

Re: Trying to reduce the number of replicates from 9 to 3

Hi Mich, You might want to take a look at this section: "Increasing replication factor" in the doc: https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor Simply put, the json file provided in kafka-reassign-partitions.sh should show the final replicas assignment after this

Re: CVEs related to Kafka

Hi Sahil, > in which version of Kafka these will be fixed https://issues.apache.org/jira/browse/KAFKA-14320 https://issues.apache.org/jira/browse/KAFKA-14107 https://issues.apache.org/jira/browse/KAFKA-14256 Maybe you can try to search the JIRA first next time. :) Thank you. Luke On Wed, May 1

Re: KRaft with even number of controllers

Hi De Gao, Usually, we won't compare the performance with even/odd cases. I think the most important reason we don't recommend using even number of nodes is the node failure tolerance. In 1 nodes case, the majority is 1, so we can't lose any node In 2 nodes case, the majority is 2, so we can't lo

Re: Kafka topic __consumer_offsets replication issue

Hi kiran, I would check the log end offset of the in-sync partition first, and check the lags of the offsets with the leader offset. (You can check by metric: `kafka.log:type=Log,name=LogEndOffset,topic=xxx, partition=xx`) Then, I would check if the follower is doing fetching without error by chec

Re: Kafka support for IPV6 only Networking stack

Hi Senjoorian, I've never tested it, but AFAIK, Kafka supports IPv6. There's also a KIP to allow Kafka to accept duplicate listener on port for IPv4/IPv6: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330 Maybe you'll be interested. Thank you. Luke On Thu, Apr 27, 2023

Re: Addition to Kafka contributor group

Hi Atul, I've added you to the contribution group in JIRA. I didn't find your account in wiki system. Let me know if you need wiki permission to create a KIP. Thank you. Luke On Fri, Apr 21, 2023 at 10:15 PM Atul Sharma wrote: > Hi Team, > > I would like to contribute to open source Kafka, can

Re: Kafka Node Shutting Down Automatically

Hi Akshay, Thanks for reporting the issue. It looks like a bug. Could you open a JIRA ticket to track it? Thank you. Luke On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar wrote: > Hello team, > >- We are using the zookeeper less Kafka (kafka Kraf

Re: About CVE-2023-25194

Hi, This is the commit to fix the CVE: https://github.com/apache/kafka/commit/ae22ec1a0ea005664439c3f45111aa34390ecaa1 2.x upgrades to 3.x includes a major version upgrade, so it'll have some compatibility issues. Please check the notable changes for v3.0 here: https://kafka.apache.org/documentati

Re: Questions about creating jira account

Hi Jimmy, You can create an account request via https://selfserve.apache.org/jira-account.html Thank you. Luke On Sat, Mar 18, 2023 at 12:03 AM zw wrote: > Hi, > It seems that JIRA access are disabled by default and I can't create an > account by myself. > > I would appreciate that if anyone c

Re: Question about KRaft

For questions related to confluent, I think you'd better ask in their channel. Luke On Fri, Mar 10, 2023 at 12:54 PM sunil chaudhari < sunilmchaudhar...@gmail.com> wrote: > Hi Luke, > This docu is good. > Does it apply for confluent as well? > > > > On Fri, 1

Re: Question about KRaft

Hi Zhenyu, Answering your question: > Should I simply 1. download 3.4 binary 2. stop ZK & Kafka service 3. upgrade Kafka to 3.4 4. start only Kafka service with KRaft server.properties That is not migrating, actually. That is just creating another kafka cluster in KRaft mode. The point for migra

Re: Kafka metrics - Requests.

Hi David, It did look like a bug. Could you file a bug in JIRA? And if you have time, welcome to investigate and submit PR for it. (My guess is there are some internal topics are included in `TotalFetchRequestsPerSec`, but not included in another one, but not sure) Thank you. Luke On Sat, Feb 25

Re: kafka-producer-perf-test.sh maximum throughput limits for broker on single node

Hi Tomasz, There are some configurations needed to be tuned in producer side. Try searching kafka producer tuning, you should find many good articles describing it. Good luck. Luke On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński < tomasz.sacin...@linuxpolska.pl> wrote: > Hello users. > > I tes

Re: Release plan for kafka-python

Hi Sahil, You should ask in kafka-python repo, not here. Thank you. Luke On Tue, Feb 14, 2023 at 7:02 PM Sahil Sharma D wrote: > Hi team, > > As per understanding its last release was released in 2020, after that > there is no release. We are planning to use v2.0.2, if we face any issue in > t

Re: Server running issue

Hi Shivannad, It said there's some exception during reading/writing data from/to /tmp/kafka-logs dir. There should be some errors in the log above these lines. That should be the root cause. Luke On Mon, Feb 13, 2023 at 11:51 PM Shivanand K Badiger wrote: > > Hi Team, > > I am facing some erro

Re: RequestsPerSec version

> Do I understand correctly that messages from 9+ do not follow the same conversion as before? No, there's no change for messages 9+. So, I think your understanding is correct. When client and server are using the same compression codec, it won't do conversion. What you need to do now is to make

Re: Re: RequestsPerSec version

vailability > "transaction.state.log.replication.factor" = 3 > "transaction.state.log.min.isr"= 1 > > # Storage Optimization > "compression.type" = "zstd" > ``` > > I tried disabling zstd, but ProduceMessageConversionsPer

Re: RequestsPerSec version

Hi Gonzalo, For the produce request record version, you should refer to this file: https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ProduceRequest.json#L35 But you're right, basically the message conversion happened in a very old produce request version (ex: ve

Re: Certificate renewal issue

Hi Sandeep, Maybe it'd be better to ask in strimzi community? Luke On Tue, Jan 10, 2023 at 5:57 PM Sandeep M wrote: > Hi all, > > We are using strimzi operator version 0.21.1 and Kubernetes 1.23.1. Broker > certs are not getting renewed as a result we are facing issue. Let me know > how to res

Re: Topic partition Leader: none

Hi, That sounds similar to this issue: KAFKA-14190 . Could you help confirm it and comment in the JIRA? That will let contributors know this is an important issue bothering many users. Thank you. Luke On Tue, Jan 10, 2023 at 10:08 AM megh vidani

Re: log.cleaner.io.max.bytes.per.second

in 97,3 seconds (14,1 Mb/sec, 53,4% of total > time) > Start size: 1 372,1 MB (10 329 108 messages) > End size: 230,0 MB (1 000 105 messages) > 83,2% size reduction (90,3% fewer messages) > > Le mer. 28 déc. 2022 à 09:41, Luke Chen a écrit : > > > H

Re: log.cleaner.io.max.bytes.per.second

Hi Nicolas, The throttler in log cleaner is just a simple throttler to control the IO speed by periodically (every 300ms) check the IO rate so far, and sleep some time to slow it down. That is, in your case, it could be: log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep to

Re: [ANNOUNCE] New committer: Josep Prat

Congratulations, Josep! Luke On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass wrote: > Congrats Josep! > > On Tue, Dec 20, 2022, 21:56 Matthias J. Sax wrote: > > > Congrats! > > > > On 12/20/22 12:01 PM, Josep Prat wrote: > > > Thank you all! > > > > > > ——— > > > Josep Prat > > > > > > Aiv

Re: KRaft? cannot recreate a particular topic

Hi Simon, This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2 and v3.4.0. Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337 Thank you Luke On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka wrote: > I am running a single node cluster in KRaft mode on >

Re: Granting permission for Create KIP and contribute to kafka

Hi t-mac, I've granted your JIRA account. But I can't find the wiki id: "ws" in the wiki system. Are your sure you registered with "ws" here ? Any more info you can provide to us, like account full name? Thank you. L

Re: Leader election strategy

Hi Pierre, Try using kafka-reassign-partitions.sh to reassign partitions to different replicas you like. ref: https://kafka.apache.org/documentation/#basic_ops_automigrate Luke On Mon, Nov 14, 2022 at 3:55 PM Pierre Coquentin wrote: > Hello, > We have a Kafka cluster (2.4.1) with a replicatio

Re: Connection issue

Hi shikha, I think you should ask in Spark community. Thanks Luke On Tue, Nov 15, 2022 at 3:17 AM shikha sharma wrote: > Hello, > > I am trying to connect to kafka using this command: > orderRawData = spark.readStream \ > .format("kafka") \ > .option("kafka.bootstrap.servers", "18.211.

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

Congrats Bruno! Well deserved! Luke On Wed, Nov 2, 2022 at 10:07 AM John Roesler wrote: > Congratulations, Bruno!!! > > On Tue, Nov 1, 2022, at 15:16, Lucas Brutschy wrote: > > Wow, congratulations! > > > > On Tue, Nov 1, 2022 at 8:55 PM Chris Egerton > wrote: > >> > >> Congrats! > >> > >> On

Re: Help with MM2 active/passive configuration

This is what Ryanne mentioned doc: https://github.com/apache/kafka/tree/trunk/connect/mirror Thanks. Luke On Mon, Oct 24, 2022 at 6:38 PM Chris Peart wrote: > > > Hi Ryanne, > > I cannot find what you mentioned, all i'm looking for are some example > configurations for and active/passive setup.

Re: in case of disk failure,why not recover from middle position of the log file

Hi Zhang, Yes, when doing log recovery, we'll start from the checkpoint for the partition in `recovery-point-offset-checkpoint` file, which will be updated at runtime. Is that what you expected? Check LogManager#loadLogs method for more implementation detail. Thanks. Luke On Thu, Oct 13, 2022 a

Re: consumer loses offset

ommit is enabled every 5 seconds? > We are checking daily and the offset is always set to NULL. > > > > Lorenzo Rovere > > Technology Reply > Via Avogadri, 2 > 31057 - Silea (TV) - ITALY > phone: +39 0422 1836521 > l.rov...@reply.it > www.reply.it > -Ori

Re: Apply to be a contributor for Kafka

Hi Divya, Done. Thanks for the interest in Apache Kafka. Luke On Mon, Oct 10, 2022 at 6:20 AM Divya A L wrote: > I’m a developer of kafka, and I want to contribute for the project. Can I > be added as a contributor, as I would like to pick up an issue from the > jira board. > > jira ID : divya

Re: consumer loses offset

Hi Lorenzo, Sounds like it is caused by this bug: https://issues.apache.org/jira/browse/KAFKA-13636 If you're not in the versions of fixed version list or newer, please try to upgrade it. Thanks. Luke On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere wrote: > Hi everyone, I have a simple question

Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

Hi Bhavesh, I understand your point. There was an old KIP with the similar idea which was not accepted by the community in the end. Maybe you can try to bring it back to the community again, or try to propose your own KIP for this idea? https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

t poll. So how does CooperativeStickyAssignore help here to > wait for process of 100 records and commit their offsets before moving the > partition to new consumer? Looks like i am missing something > > On Fri, Sep 16, 2022 at 7:59 AM Luke Chen wrote: > > > Hi Pushkar, > > &

Re: CVE-2022-34917: Unauthenticated clients may cause OutOfMemoryError on Apache Kafka Brokers

What a great finding! Thanks Mickael Maison, Tom Bentley and Daniel Collins! And thanks for all the release managers who help drive all these security patch releases! Luke On Mon, Sep 19, 2022 at 11:53 PM Manikumar wrote: > Severity: High > > Description: > > A security vulnerability has been

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

Hi Pushkar, Here's the answer to your questions: > 1. During scale-down operation, I am adding a shutdown hook to the Java Runtime, and calling close on the consumer. As per kafka docs, close provides 30 sec to commit current offsets if auto.commit is enabled: so, i assume that it will process th

Re: Add to kafka contributor list

Hi Vinay, Done. Thanks for the interest in Apache Kafka. Luke On Mon, Sep 5, 2022 at 1:42 PM vinay kumar wrote: > Hi Bill, > > My jira user : vvakati > > Can you add me to contributor list please? > > Thanks, > Vinay > > On Fri, 2 Sep 2022 at 12:09 AM, Bill Bejeck > wrote: > > > Hi Vinay, > >

  1   2   3   >