Re: Redis as state store

Thanks Sophie... that answers my question, however still worried about some other aspects: 1. If redis is to be restored from changelog topic: what would happen if i have 3 stream applications and 1 instance went down ... will other 2 instances halt until entire existing state from redis is wiped

Re: Kafka Streams And Partitioning

Ah ok, I think I was envisioning a different use case from your initial description of the problem. If everything that you want to group together is already correctly partitioned, then you won't need a repartitioning step. If I understand correctly, you have something like this in mind: builder

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hey Mickael, I just merged the fix back to 2.7 so you should be good to go Thanks for the PR Bruno! On Fri, Mar 19, 2021 at 9:34 AM Mickael Maison wrote: > Thanks Bruno, > > That indeed sounds like a blocker. > > I'm closing this vote, I'll build a new RC once a fix is merged into 2.7 > > On Fr

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Thanks Bruno. I agree this qualifies as a blocker since it was a regression in 2.6 and may result in data loss. I'll roll a new RC with the fix On Fri, Mar 19, 2021 at 7:03 AM 'Bruno Cadonna' via kafka-clients < kafka-clie...@googlegroups.com> wrote: > Hi Sophie, > > Correction to my last e-mail:

Re: Redis as state store

Hey Pushkar, yes, the data will still be backed by a changelog topic unless the user explicitly disables logging for that state store. The fault tolerance mechanism of Kafka Streams is based on changelogging, therefore there are no correctness guarantees if you decide to disable it. That said, I'm

Re: Redis as state store

Hello Sophie, may be i am missing something here, however can you let me know how a redis based state store will be wiped off the inconsistent state in case stream application dies in the middle of processing e.g. stream application consumed from source topic, processed source event and saved stat

Re: Kafka SLIs/SLOs

Any input in this is appreciated... On Tue, Mar 9, 2021, 7:19 AM rs vas wrote: > I am looking for defining these Availability, Latency, ErrorRate, Load for > Kafka clusters using Prometheus metrics generated by KAfka and jmx > exporter. > > Does anyone have any initial dashboards to start with f

Re: does Kafka exactly-once guarantee also apply to kafka state stores?

Matthias, With reference to your response above, i came across the JIRA ticket https://issues.apache.org/jira/browse/KAFKA-12475 For rocksDB or in-memory state stores, these are always backed by changelog topic, so they can be rebuilt from scratch from the changelog topic. However, how a remote s

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Thanks Bruno, That indeed sounds like a blocker. I'm closing this vote, I'll build a new RC once a fix is merged into 2.7 On Fri, Mar 19, 2021 at 2:04 PM Bruno Cadonna wrote: > > Hi Mickael, > > Correction to my last e-mail: The bug does not break eos, but it breaks > at-least-once. > > Bruno >

Re: Kafka connect replication using MirrorMaker 2.0

Hello and thank you for the reply! My problem is not with consumption of messages, because as you said, MirrorMaker2 knows how to deal with the consumer offsets. Rather my problem is with source connectors and the topic connect-offsets. Because Kafka connect manages where it stopped reading f

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker sin

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a blocker.

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a blocker

Re: [VOTE] 2.6.2 RC0

Hi Sophie, +1 binding - checked signatures - built from source - verified quickstart with Scala 2.13 binaries - checked javadocs Thanks for running the release On Fri, Mar 19, 2021 at 10:13 AM Manikumar wrote: > > Hi, > > +1 (binding) > > - verified the signatures > - ran the tests on the sour

[VOTE] 2.7.1 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.7.1. Apache Kafka 2.7.1 is a bugfix release and 40 issues have been fixed since 2.7.0. Release notes for the 2.7.1 release: https://home.apache.org/~mimaison/kafka-2.7.1-rc0/RELEASE_NOT

Re: [VOTE] 2.6.2 RC0

Hi, +1 (binding) - verified the signatures - ran the tests on the source archive with Scala 2.13 - verified the quickstart with Scala 2.12 binary - verified the artifacts, javadoc Thanks for running the release! Thanks, Manikumar On Tue, Mar 16, 2021 at 9:41 AM Sophie Blee-Goldman wrote: > T

Re: Right Use Case For Kafka Streams?

just my 2 cents the best answer is always from the real-world practices :) RocksDB https://rocksdb.org/ is the implementation of "state store" in Kafka Stream and it is an "embedded" kv store (which is diff than the distributed kv store). The "state store" in Kafka Stream is also backed up by "

Re: Kafka connect replication using MirrorMaker 2.0

Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the process of consumption. just my 2 cents - If you already have two Kafka connect clusters in two different sites, it sounds practical to: (1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2 (2) add one "MirrorMak