Re: Add user to contributors list

Hi, no, for that you need to become a committer. Infos to get to committership can be found under the following link: https://kafka.apache.org/contributing tl;dr You need to contribute to Apache Kafka, help the community, and demonstrate good knowledge of the project. If this sounds appeal

Re: [ANNOUNCE] New committer: Apoorv Mittal

Well deserved! Congrats! Best, Bruno On 10.04.25 18:02, Sanskar Jhajharia wrote: Congratulations Apoorv! On Thu, Apr 10, 2025 at 9:32 PM Lucas Brutschy wrote: Congrats! Well deserved, Apoorv! On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai wrote: Welcome Apoorv as a committer! You des

Re: [ANNOUNCE] New committer: TengYao Chi

Well deserved! Congrats! Best, Bruno On 10.04.25 18:01, Lucas Brutschy wrote: Congrats! On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai wrote: You are the best! TengYao Matthias J. Sax 於 2025年4月10日 週四 下午11:55寫道： The Project Management Committee (PMC) for Apache Kafka is pleased to annou

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 4.0.0

, bachmanity1, bboyleonp666, Bill Bejeck, brenden20, Bruno Cadonna, Caio Guedes, Calvin Liu, Chengyan, Cheryl Simmons, Chia-Chuan Yu, Chia-Ping Tsai, Chirag Wadhwa, Chris Egerton, Christo Lolov, Christopher L. Shannon, ClarkChen, Clay Johnson, Colin P. McCabe, Colt McNealy, Danica Fine, Dániel Urbán

Re: Optimizing Kafka Streams Validation for Array Elements

e":2, "timestamp":"2024-04-08T02:06:49.000Z" } } And perfect soultion should use only two topics source and sink (no additional repartition and changelog), and as I mentioned before we don't like to introduce manual validation in the flatMap (like if null

Re: Optimizing Kafka Streams Validation for Array Elements

Hi Pawel, I am not completely sure I understand the issue, because I am not a JSON expert. Is my understanding correct that the serde that you use to read from the input topic and from the repartition topic also do the validation of the JSON? Regarding point 2 and 3: I agree that the depen

Re: 回复: Kafka 3.8.1 /tmp exec permission issue also has impact on Kafka client

Hi Benson, you got a reply from Josep on Feb 17th. Copying Josep's e-mail here for simplicity: Hi Xiong, The release notes are also in Github (both github.com/apache/kafka/ and github.com/apache/kafka-site) and we can fix them with a PR. Regarding your findings, what you say makes sense, and i

Re: [ANNOUNCE] New committer: Lianet Magrans

Well deserved! Best, Bruno On 8/28/24 6:37 PM, Bill Bejeck wrote: Congrats Lianet! On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax wrote: Congrats! Very well deserved! On 8/28/24 9:18 AM, Justine Olshan wrote: Congratulations Lianet!! Justine On Wed, Aug 28, 2024 at 8:58 AM David Arthur

Re: Franz Kafka 100th

+1 Thanks for the message! Best, Bruno On 6/3/24 2:50 PM, Edoardo Comar wrote: Hi all, For an Apache project that bear his name, I think it’s fitting to remember that today Monday 3rd it’s the 100th anniversary of Franz’s death. https://www.ox.ac.uk/news/2024-03-04-oxford-kafka-2024 Unless

Re: Can two different kafka clusters be used in kafka streams for consumption from 1 cluster and produce on another cluster

Hi Pushkar, unfortunately, cross cluster processing is currently not possible with Kafka Streams. Best, Bruno On 4/11/24 4:13 PM, Pushkar Deole wrote: Hi All, We are using a streams application and currently the application uses a common kafka cluster that is shared along with many other ap

Re: Messages disappearing from Kafka Streams topology

Hi, That sounds worrisome! Could you please provide us with a minimal example that shows the issue you describe? That would help a lot! Best, Bruno On 3/25/24 4:07 PM, Karsten Stöckmann wrote: Hi, are there circumstances that might lead to messages silently (i.e. without any logged warnin

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

. Kind regards, Venkatesh From: Bruno Cadonna Date: Friday, 15 March 2024 at 8:47 PM To: users@kafka.apache.org Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled Hi Venkatesh, As you discovered, in Kafka Streams 3.5.1 there is no stop-the-world rebalancing. Static group

Re: Joining the Kafka Users Mailing List

Hi Vansh, Great that you want to join our community! Subscription to the mailing list is self-serve. See details to subscribe under the following link: https://kafka.apache.org/contact Thank you for your interest in Apache Kafka! Best, Bruno On 3/15/24 1:59 PM, Vansh Kabra wrote: Dear Ka

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

I can use which can make offset lags observable even during rebalancing, can you please let me know? Thank you very much. Kind regards, Venkatesh From: Bruno Cadonna Date: Wednesday, 13 March 2024 at 8:29 PM To: users@kafka.apache.org Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app se

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

Hi Venkatesh, Extending on what Matthias replied, a metadata refresh might trigger a rebalance if the metadata changed. However, a metadata refresh that does not show a change in the metadata will not trigger a rebalance. In this context, i.e., config METADATA_MAX_AGE_CONFIG, the metadata is t

Re: [ANNOUNCE] Apache Kafka 3.7.0

, Arpit Goyal, Artem Livshits, Ashwin Pankaj, ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna, Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya

Re: Question about using public classes from the internal packages

Hi John, In general, we do not guarantee anything on APIs of the internal package. That is also the reason why you do not need a KIP to change those classes. Any class for which the build generates Javadoc is considered public API [1]. For public APIs we guarantee backwards compatibility. B

Re: Fwd: unsubscribe

Hi Clark, Unfortunately, I cannot verify your subscription but a mistake that often happens is that people try to unsubscribe from the mailing list with an e-mail address that is different from the one with which they subscribed to the mailing list. Please verify that you use the e-mail addre

Re: License Cost for Apache Kafka with Support

Hi Vinothkumar, The Apache Kafka project and the Apache Software Foundation do not sell licenses for Apache Kafka. Apache Kafka is distributed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). This is an open source license that gives you quite some freedom. Neither

Re: [ANNOUNCE] Apache Kafka 3.6.0

, Bruno Cadonna, Calvin Liu, Chaitanya Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190, Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur, David Jacot, David Mao, dengziming, Deqi Hu, Dimitar

Re: adding enum value in kafka streams

Hi Mariusz, How is fooKey de-/serialized? I ask that because maybe the serializer for fooKey cannot handle the extended enum. Best, Bruno On 9/20/23 12:22 AM, M M wrote: Hello, This is my first time asking a question on a mailing list, so please forgive me any inaccuracies. I am having a

Re: Kafka Streams, read standby time window store

StaleStore(); /* */}/* and now it works well! thanks a lot for your help! On 9/6/23 16:05, Bruno Cadonna wrote: Hi Igor, Sorry to hear you have issues with querying standbys! I have two questions to clarify the situation: 1. Did you enable querying stale stores with StoreQueryParameters.fromN

Re: Kafka Streams, read standby time window store

Hi Igor, Sorry to hear you have issues with querying standbys! I have two questions to clarify the situation: 1. Did you enable querying stale stores with StoreQueryParameters.fromNameAndType(TABLE_NAME, queryableStoreType).enableStaleStores() as described in the blog post? 2. Since you a

Re: Impact and fix of CVE-2023-34462 and CVE-2023-35116

Hi Sahil, we value that you raise security issues. However, since this is a sensitive area there are guidelines to follow as François pointed out. Additionally to what François said, I would like to remind you again (as I did in https://kafka.apache.org/project-security) to follow the Kafka-

Re: Release plan required for version 3.5.1

Hi Sahil, Please have a look at the dependencies for Kafka 3.5.1: https://github.com/apache/kafka/blob/3.5.1/gradle/dependencies.gradle and compare it with your list of CVEs. Please also have a look here: https://kafka.apache.org/project-security If you discover a security issue please follow

Re: kafka streams re-partitioning on incoming events

hat we normally store in Processor in Topology? On Fri, Jul 14, 2023 at 3:50 PM Bruno Cadonna wrote: Hi Pushkar, you can use repartition() for repartition your data. Method through() is actually deprecated in favor of repartition(). Before you repartition you need to specify the new key with sel

Re: kafka streams re-partitioning on incoming events

Hi Pushkar, you can use repartition() for repartition your data. Method through() is actually deprecated in favor of repartition(). Before you repartition you need to specify the new key with selectKey(). What do you mean exactly with "...and then process them in that order"? The order of th

Re: [ANNOUNCE] New committer: Greg Harris

Congratulations Greg! Best, Bruno On 10.07.23 17:47, Matthias J. Sax wrote: Congrats! On 7/10/23 8:45 AM, Chris Egerton wrote: Hi all, The PMC for Apache Kafka has invited Greg Harris to become a committer, and we are happy to announce that he has accepted! Greg has been contributing to K

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.5.0

Mickael, Thanks for driving the release! Best, Bruno On 15.06.23 10:27, Mickael Maison wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.5.0. This is a minor release and it includes fixes and improvements from 201 JIRAs. All of the changes in this rel

Re: Kafka Stream: Cannot invoke "org.apache.kafka.streams.processor.internals.SourceNode.getTimestampExtractor()" because "source" is null

Hi Andrew, what version of Kafka Streams do you use? Since 2.7 there is a null check for the source node [1]. The following ticket might be related: https://issues.apache.org/jira/browse/KAFKA-10205 Best, Bruno [1] https://github.com/apache/kafka/blob/20028e24cca91422b8f02fdbf45d2cd9ef24c9

Re: [VOTE] 3.4.1 RC3

Hi Prem, You need to remove yourself from the mailing list. Please find the details to do so here: https://kafka.apache.org/contact The interesting part for you is this: "To unsubscribe, send an email to users-unsubscr...@kafka.apache.org" Maybe you should also check if you are subscribed to t

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

Congrats, David! Bruno On 10.03.23 01:36, Matthias J. Sax wrote: Congrats! On 3/9/23 2:59 PM, José Armando García Sancio wrote: Congrats David! On Thu, Mar 9, 2023 at 2:00 PM Kowshik Prakasam wrote: Congrats David! On Thu, Mar 9, 2023 at 12:09 PM Lucas Brutschy wrote: Congratulations

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

Thank you all! I am really honoured to have become a PMC member. Looking forward to contributing further to Apache Kafka. Best, Bruno On 02.11.22 17:02, Levani Kokhreidze wrote: Congrats Bruno, well deserved! On Wed, 2 Nov 2022, 11:11 Jorge Esteban Quilcate Otoya, < quilcate.jo...@gmail.com>

Re: Commercial support

28/09/2022 11:05, Bruno Cadonna wrote: Hi Francois, I am not aware of such a page on the Apache Kafka website. There are a variety of companies that sell Kafka as a self-hosted platform or as a Cloud-hosted service. Those companies include Confluent (disclaimer: I work for them), Red Hat, AWS,

Re: Commercial support

Hi Francois, I am not aware of such a page on the Apache Kafka website. There are a variety of companies that sell Kafka as a self-hosted platform or as a Cloud-hosted service. Those companies include Confluent (disclaimer: I work for them), Red Hat, AWS, Aiven, Instaclustr, Cloudera, and mo

Re: Kafka stream moving from running to shutdown without any visible exception or error

Hi Samuel, from your few log messages I see that the global stream thread dies. That might be the cause of the shutdown. Look for what is causing your global stream thread/global store to die. Best, Bruno On 13.09.22 22:55, Samuel Azcona wrote: Hi guys, I have a kotlin App that run 5 str

Re: Apache atmosphere is sending an empty message every 5 minutes to the north bound. These messages contain a single byte, which is represented as a space character ' '. Is there any ability to disab

Hi Ori, I think you wrote your e-mail to the wrong list. This is the Apache Kafka user mailing list. Best, Bruno On 13.09.22 09:13, fcb man wrote: Hi Guys, We are using Apache Atmosphere and several users shared the same concern, where the north bound client (netcool in their case) is gettin

Re: KAFKA 3.2.1 Vulnerabilities fix required

-Original Message- From: Bruno Cadonna Sent: 09 September 2022 03:15 PM To: users@kafka.apache.org Subject: Re: KAFKA 3.2.1 Vulnerabilities fix required Hi Sahil, I do not know when the next 3.2 patch release will be out. However, the upcoming 3.3.0 release fixes these vulnerabilities. The

Re: KAFKA 3.2.1 Vulnerabilities fix required

Hi Sahil, I do not know when the next 3.2 patch release will be out. However, the upcoming 3.3.0 release fixes these vulnerabilities. The release candidate 1 of the 3.3.0 release is currently in the voting phase. Best, Bruno On 09.09.22 11:33, Sahil Sharma D wrote: Hello team, We are plann

Re: [ANNOUNCE] Apache Kafka 3.2.1

, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 19 contributors to this release! Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang

Re: [ANNOUNCE] Apache Kafka 3.2.0

we fix that typo? Thanks! -James On May 17, 2022, at 9:01 AM, Bruno Cadonna wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.2.0 * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the partiti

Re: Leverage multiple disks for kafka streams stores

itrarily attach multiple volumes, sometimes it is cheaper to use several cheaper volumes in parallel than a single very expensive one. I personally believe that this should be considered for a KIP. Best regards, Adrian Tubio On Thu, May 19, 2022 at 1:49 PM Bruno Cadonna wrote: Hi Adrian, I am

Re: Leverage multiple disks for kafka streams stores

Hi Adrian, I am afraid that you cannot set the state directory for a single state store to a different directory than all other stores. Maybe the following blog post can help you debug and solve your issue: https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performan

[ANNOUNCE] Apache Kafka 3.2.0

Khamphousone, bozhao12, Bruno Cadonna, Chang, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe, Colin Patrick McCabe, Cong Ding, David Arthur, David Jacot, David Mao, defhacks, dengziming, Ed B, Edwin, florin-akermann, GauthamM-official, GuoPhilipse, Guozhang Wang, Hao Li, Haoze Wu, Idan Kamara

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

t 10:36 AM Jakub Scholz wrote: +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven artifacts and ran various tests with them. Thanks for doing the release. Jakub On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is

[RESULTS] [VOTE] Release Kafka version 3.2.0

This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Bill Bejeck * Mickael Maison * David Jacot Committers: * Luke Chen Community: * Jakub Scholz 0 votes * No votes -1 votes * No votes Vote thread: https://lists.apache.org/thread/3b05b74hw7cf4p2o1dn5n33

Re: [VOTE] 3.2.0 RC1

Hello Kafka users, developers and client-developers, I've got a green build with unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/45/ Best, Bruno On 03.05.22 16:07, Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is the s

[VOTE] 3.2.0 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the partition leader to recover the partition (KIP-704) * Top-level er

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC0

.2.0? This fixes an issue introduced in 3.2.0 where in some of the error cases, the producer interceptor is called twice for the same record. Thanks, Jun On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna <mailto:cado...@apache.org>> wrote: Hi all, This is a gently remin

Re: [VOTE] 3.2.0 RC0

://jenkins.confluent.io/job/system-test-kafka/job/3.2/24/ Thank you to Michal for voting on the release candidate. Best, Bruno On 15.04.22 21:05, Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced with

Re: Partnership: Apache Kafka & SantoDigital

Hi Amanda, Apache Kafka is an open source project. You can download, install, and use the software for free. Some companies offer Apache Kafka as a cloud service or as an on-prem software. You can find those companies by asking google. Best, Bruno On 25.04.22 17:11, Amanda Jampaulo wrote

[VOTE] 3.2.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the partition leader to recover the partition (KIP-704) * Top-level err

Re: Need Help - getting vulnerability due to Log4j- v1.2.17 jar being used in Kafka_2.11-2.4.0.

Hi Sandip, I just merged the PR https://github.com/apache/kafka/pull/11743 that replaces log4j with reload4j. Reload4j will be part of Apache Kafka 3.2.0 and 3.1.1. Best, Bruno On 30.03.22 04:26, Luke Chen wrote: Hi Sandip, We plan to replace log4j with reload4j in v3.2.0 and v3.1.1. (KAFK

Re: Setting up the CooperativeStickyAssignor in Java

Hi Richard, The group.instance.id config is orthogonal to the partition assignment strategy. The group.instance.id is used if you want to have static membership which is not related to the partition assignment strategy. If you think you found a bug, could you please open a JIRA ticket with s

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.0.1

, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 26 contributors to this release! A. Sophie Blee-Goldman, Andras Katona, Bruno Cadonna, Chris Egerton, Cong Ding, David Jacot, dengziming, Edoardo Comar, Ismael Juma

Re: Reducing issue

Hi Robin, since this seems to be a ksql question, you will more likely get an answer here: https://forum.confluent.io/c/ksqldb Best, Bruno On 01.02.22 10:03, Robin Helgelin wrote: Hi, Working on a small MVP and keep running into a dead end when it comes to reducing data. Began using ksql

Re: Improving I/O on KafkaStreams

Hi Murilo, Have you checked out the following blog post on tuning performance of RocksDB state stores [1] especially the section on high disk I/O and write stalls [2]? Do you manage the off-heap memory used by RocksDB as described in the Streams docs [3]? I do not know what may have caused

Re: `java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS` after upgrading `Kafka-clients` from 2.5.0 to 3.0.0

Hi Bruce, I do not know the specific root cause of your errors but what I found is that Spring 2.7.x is compatible with clients 2.7.0 and 2.8.0, not with 3.0.0 and 2.8.1: https://spring.io/projects/spring-kafka Best. Bruno On 24.09.21 00:25, Chang Liu wrote: Hi Kafka users, I start runnin

Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD

: Hi Bruno, thank you for your answer. I mean that the message that caused the exception was consumed and replaced thread will continue from the next message. How then does it handle uncaught exceptions, if it will fail again? On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna wrote: Hi Yoda, What

Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD

Hi Yoda, What do you mean exactly with "skipping that failed message"? Do you mean a record consumed from a topic that caused an exception that killed the stream thread? If the record killed the stream thread due to an exception, for example, a deserialization exception, it will probably als

Re: disappearing messages with kafka streams

Hi Günter, What is the timestamp of the records? The difference between the system time on the brokers and the record timestamp is used to decide whether a record segment should be removed because its retention time is exceeded. So if the retention time of the topic is set to 1.5 days, the re

Re: command to get the value of a config from the broker

Hi Dhirendra, You could use the kafka-configs.sh script or in Java the admin client (see https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/admin/Admin.html) Best, Bruno On 01.07.21 13:45, Dhirendra Singh wrote: Hi All, I want to get the value of a config from broker. I do not hav

Re: Issue with StreamsBuilder.stream(Pattern)

Hi Will, This looks like a bug to me. Could you please open a Jira with the stacktrace of the exception and a minimal repro example? Best, Bruno On 08.06.21 16:51, Will Bartlett wrote: Hi all, I'm hitting a NPE in a very basic repro. It happens when toString() is called on the StreamSource

Re: Please add user heyingquan to the list of contributors

Hi, I added you to the list of contributors in the Apache Kafka JIRA project. You can now assign tickets to yourself. Welcome to Apache Kafka! Best, Bruno On 05.06.21 15:44, 和映泉 wrote: Please add user heyingquan to the list of contributors.

Re: Streams Partition Allocation Skewness

Hi Navneeth, I need some clarifications to be able to help you. First of all it would be useful to know if your topology is stateful, i.e., if it has to maintain state. Since you have two topics with 72 partitions but only 72 tasks (or partitions groups to assign) that need to be distributed

Re: State Store Data Retention

correct? Thanks On Mon, Apr 19, 2021 at 1:57 AM Bruno Cadonna wrote: Hi Upesh, The answers to your questions are: 1. The configs cleanup.policy and retention.ms are topic configs. Hence, they only affect the changelog of a state store, not the local state store in a Kafka Streams client

Re: Changing Replication Factor

Hi Marcus, 1. If you change REPLICATION_FACTOR_CONFIG without resetting the application (or deleting the changelog and repartition topics) and redeploy the Streams application, the replication factor of the internal topics will not change. The replication factor will only change for new deplo

Re: Kafka Streams and retention limits

Additionally, with KIP-698 (https://cwiki.apache.org/confluence/x/7CnZCQ), we will add verifications of changelogs to find misconfigurations and report them to the users. Best, Bruno On 24.04.21 22:58, Matthias J. Sax wrote: The topics used by Kafka Streams to backup state stores, are calle

Re: State Store Data Retention

Hi Upesh, The answers to your questions are: 1. The configs cleanup.policy and retention.ms are topic configs. Hence, they only affect the changelog of a state store, not the local state store in a Kafka Streams client. Locally, window and session stores remove data they do not need anymore.

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

Congrats Randall! Well deserved! Bruno On 17.04.21 01:43, Matthias J. Sax wrote: Hi, It's my pleasure to announce that Randall Hauch in now a member of the Kafka PMC. Randall has been a Kafka committer since Feb 2019. He has remained active in the community since becoming a committer. Cong

Re: [ANNOUNCE] New Committer: Bruno Cadonna

Thank you all for the kind words! Best, Bruno On 08.04.21 00:34, Guozhang Wang wrote: Hello all, I'm happy to announce that Bruno Cadonna has accepted his invitation to become an Apache Kafka committer. Bruno has been contributing to Kafka since Jan. 2019 and has made 99 commits and

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

Congrats Bill! Well deserved! Best, Bruno On 12.04.21 11:19, Satish Duggana wrote: Congratulations Bill!! On Thu, 8 Apr 2021 at 13:24, Tom Bentley wrote: Congratulations Bill! On Thu, Apr 8, 2021 at 2:36 AM Luke Chen wrote: Congratulations Bill! Luke On Thu, Apr 8, 2021 at 9:17 AM Mat

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a blocker.

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a blocker

Re: [ANNOUNCE] New committer: Tom Bentley

Congrats, Tom! Best, Bruno On 15.03.21 18:59, Mickael Maison wrote: Hi all, The PMC for Apache Kafka has invited Tom Bentley as a committer, and we are excited to announce that he accepted! Tom first contributed to Apache Kafka in June 2017 and has been actively contributing since February 20

Re: Error upgrading KafkaStreams

5 Mar 2021 at 10:20, Bruno Cadonna wrote: Hi Murilo, A couple of questions: 1. What do you mean exactly with clean up? 2. Do you have acleanup policy specified on the changelog topics? Best, Bruno On 15.03.21 15:03, Murilo Tavares wrote: Hi Bruno No, I haven't tested resetting the a

Re: Redis as state store

Hi Alex, I guess wiping out the state directory is easier code-wise, faster, and/or at the time of development the developers did not design for remote state stores. But I do actually not know the exact reason. Off the top of my head, I do not know how to solve this for remote state stores.

Re: Error upgrading KafkaStreams

Clean up and upgrade to 2.7. No error this time. Thanks Murilo On Mon, 15 Mar 2021 at 09:53, Bruno Cadonna wrote: Hi Murilo, Did you retry to upgrade again after you reset the application? Did it work? Best, Bruno On 15.03.21 14:26, Murilo Tavares wrote: Hi Bruno Thanks for your response.

Re: Error upgrading KafkaStreams

Murilo On Mon, 15 Mar 2021 at 06:21, Bruno Cadonna wrote: Hi Murilo, No, you do not need any special procedure to upgrade from 2.4 to 2.7. What you see in the logs is not an error but a warning. It should not block you on startup forever. The warning says that the local states of task 7_17 are

Re: Redis as state store

Hi Alex, You are right! There is no "exactly once magic" backed into the RocksDB store code. The point is local vs remote. When a Kafka Streams client closes dirty under EOS, the state (i.e., the content of the state store) needs to be wiped out and to be re-created from scratch from the chan

Re: Error upgrading KafkaStreams

Hi Murilo, No, you do not need any special procedure to upgrade from 2.4 to 2.7. What you see in the logs is not an error but a warning. It should not block you on startup forever. The warning says that the local states of task 7_17 are corrupted because the offset you want to fetch of the st

Re: RocksDB state store disk space estimation

Hi Chris, your estimation looks correct to me. I do not know how big M might be. Maybe the following link can help you with the estimation: https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format There are also some additional files that RocksDB keeps in its directory. I gue

Re: kafka stream processor's process method

Hi Sathya, MyProcessor does not have access to MySource, because in MySource you just build the topology that is then executed by Kafka Streams. So you cannot send anything to MySource, because MyProcessor does not know anything about MySource. If you want to stop consumption upon an excepti

Re: Kafka streams - how to handle application level exception in event processing

unding Circle Engineering On Tue, 22 Sep 2020 at 08:12, Bruno Cadonna wrote: Hi Pushkar, I think there is a misunderstanding. If a consumer polls from a partition, it will always poll the next event independently whether the offset was committed or not. Committed offsets are used for faul

Re: Kafka streams - how to handle application level exception in event processing

streams internally? On Mon, Sep 21, 2020 at 9:01 PM Bruno Cadonna wrote: Hi Pushkar, If you want to keep the order, you could still use the state store I suggested in my previous e-mail and implement a queue on top of it. For that you need to put the events into the store with a key that

Re: Kafka streams - how to handle application level exception in event processing

ame event until the other application starts up and required data becomes available in globalKtable On Mon, Sep 21, 2020 at 5:42 PM Bruno Cadonna wrote: Thank you for clarifying! Now, I think I understand. You could put events for which required data in the global table is not available int

Re: kafka schema registry - some queries and questions

Hi Pushkar, This question is better suited for https://groups.google.com/g/confluent-platform since the Schema Registry is part of the Confluent Platform but not of Apache Kafka. Best, Bruno On 21.09.20 16:58, Pushkar Deole wrote: Hi All, Wanted to understand a bit more on the schema regis

Re: Kafka streams - how to handle application level exception in event processing

level error e.g. say, some data required for processing an input event is not available in the GlobalKTable since it is not yet synced with the global topic On Mon, Sep 21, 2020 at 4:54 PM Bruno Cadonna wrote: Hi Pushkar, Is the error you are talking about, one that is thrown by Kafka Streams or

Re: Kafka streams - how to handle application level exception in event processing

Hi Pushkar, Is the error you are talking about, one that is thrown by Kafka Streams or by your application? If it is thrown by Kafka Streams, could you please post the error? I do not completely understand what you are trying to achieve, but maybe max.task.idle.ms [1] is the configuration yo

Re: Handle exception in kafka stream

Hi Deepak, Do you return DeserializationHandlerResponse.CONTINUE or DeserializationHandlerResponse.FAIL in your CustomExceptionHandler? With DeserializationHandlerResponse.CONTINUE, the processing of records should not stop and after the next offset commit the bad records should not be read

Re: Kafka streams sink outputs weird records

Hi Pirow, hard to to have an idea without seeing the code that is executed in the processors. Could you please post a minimal example that reproduces the issue? Best, Bruno On 20.08.20 14:53, Pirow Engelbrecht wrote: Hello, I’ve got Kafka Streams up and running with the following topology:

Re: Sharing of State Stores

Hi Charles, Two transformers that share the same state store should end up into the same sub-topology. A sub-topology is executed by as many task as the number of partitions of the input topics. Each task processes the records from one input partition group (i.e. the same partition from both

Re: Question regarding Kafka Streams Global State Store

t; > Tel. +49 711 811-49893 > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > -Ursprüngliche Nachricht- > Von: Bruno Cadonna > Gesendet: Dienstag, 19. Mai 2020 11:42 > An: Users > Betreff: Re: Question regarding Kafka Streams Global State Store > > Hi Georg, >

Re: Question regarding Kafka Streams Global State Store

> > Georg Schmidt-Dumont > BCI/ESW17 > Bosch Connected Industry > > Tel. +49 711 811-49893 > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > -Ursprüngliche Nachricht- > Von: Bruno Cadonna > Gesendet: Dienstag, 19. Mai 2020 10:52 > An: Users

Re: Question regarding Kafka Streams Global State Store

Hi Georg, >From your description, I do not see why you need to use a global state instead of a local one. Are there any specific reasons for that? With a local state store you would have the previous record immediately available. Best, Bruno On Tue, May 19, 2020 at 10:23 AM Schmidt-Dumont Georg

Re: About Kafka Stream parallelism

Hi Rapeepat, 1. The parallelism of Kafka Streams does not only depend on the number of partitions of the input topic. It also depends on the structure of your topology. Your example topology topicA => transform1 => topicB => transform2 => topicC would be subdivided in two subtopologies: - subtopo

Re: KafkaStream groupBy + count on KTable behaviour

chanism should be a > bit more transparent, but it aslo maybe that I'm plain wrong here :) > > Thanks ! > > On Thu, May 14, 2020 at 9:24 PM Bruno Cadonna wrote: > > > Hi Raffaele, > > > > Change is an internal class in Streams and also its SerDes are > >

Re: KafkaStream groupBy + count on KTable behaviour

> > Thanks, > > One more thing, As I told you I was consuming the repartitioning topic > > created by group by > > and I just saw the old and new value, as you are telling me now they are > > indeed marked as old and new, > > is this mark visible somehow consuming

  1   2   >