Bruno, i tried to explain this in 'kafka user's language through above mentioned scenario, hope i put it properly -:) and correct me if i am wrong
On Mon, Mar 15, 2021 at 7:23 PM Pushkar Deole <pdeole2...@gmail.com> wrote: > This is what I understand could be the issue with external state store: > > kafka stream application consumes source topic, does processing, stores > state to kafka state store (this is backed by topic) and before producing > event on destination topic, the application fails with some issue. In > this case, the transaction has failed, so kafka guarantees either all or > none, means offset written to source topic, state written to state store > topic, output produced on destination topic... all of these happen or none > of these and in this failure scenario it is none of these. > > Assume you have redis state store, and you updated the state into redis > and stream application failed. Now, you have source topic and destination > topic consistent i.e. offset is not committed to source topic and output > not produced on destination topic, but you redis state store is > inconsistent with that since it is external state store and kafka can't > guarantee rollback ot state written there > > On Mon, Mar 15, 2021 at 6:30 PM Alex Craig <alexcrai...@gmail.com> wrote: > >> " Another issue with 3rd party state stores could be violation of >> exactly-once guarantee provided by kafka streams in the event of a failure >> of streams application instance" >> >> I've heard this before but would love to know more about how a custom >> state >> store would be at any greater risk than RocksDB as far as exactly-once >> guarantees are concerned. They all implement the same interface, so as >> long as you're correctly implementing get(), put(), delete(), flush(), >> etc, >> you should be fine right? In other words, I don't think there is any >> special "exactly once magic" that is baked into the RocksDB store code. I >> could be wrong though so I'd love to hear people's thoughts, thanks, >> >> Alex C >> >> On Sun, Mar 14, 2021 at 4:58 PM Parthasarathy, Mohan <mpart...@hpe.com> >> wrote: >> >> > Thanks for the responses. In the worst case, I might have to keep both >> > rocksdb for local store and keep an external store like Redis. >> > >> > -mohan >> > >> > >> > On 3/13/21, 8:53 PM, "Pushkar Deole" <pdeole2...@gmail.com> wrote: >> > >> > Another issue with 3rd party state stores could be violation of >> > exactly-once guarantee provided by kafka streams in the event of a >> > failure >> > of streams application instance. >> > Kafka provides exactly once guarantee for consumer -> process -> >> > produce >> > through transactions and with the use of state store like redis, >> this >> > guarantee is weaker >> > >> > On Sat, Mar 13, 2021 at 5:28 AM Guozhang Wang <wangg...@gmail.com> >> > wrote: >> > >> > > Hello Mohan, >> > > >> > > I think what you had in mind works with Redis, since it is a >> remote >> > state >> > > store engine, it does not have the co-partitioning requirements as >> > local >> > > state stores. >> > > >> > > One thing you'd need to tune KS though is that with remote stores, >> > the >> > > processing latency may be larger, and since Kafka Streams process >> all >> > > records of a single partition in order, synchronously, you may >> need >> > to tune >> > > the poll interval configs etc to make sure KS would stay in the >> > consumer >> > > group and not trigger unnecessary rebalances. >> > > >> > > Guozhang >> > > >> > > On Thu, Mar 11, 2021 at 6:41 PM Parthasarathy, Mohan < >> > mpart...@hpe.com> >> > > wrote: >> > > >> > > > Hi, >> > > > >> > > > I have a use case where messages come in with some key gets >> > assigned some >> > > > partition and the state gets created. Later, key changes (but >> still >> > > > contains the old key in the message) and gets sent to a >> different >> > > > partition. I want to be able to grab the old state using the old >> > key >> > > before >> > > > creating the new state on this instance. Redis as a state store >> > makes it >> > > > easy to implement this where I can simply do a lookup before >> > creating the >> > > > state. I see an implementation here : >> > > > >> > > >> > >> https://github.com/andreas-schroeder/redisks/tree/master/src/main/java/com/github/andreas_schroeder/redisks >> > > > >> > > > Has anyone tried this ? Any caveats. >> > > > >> > > > Thanks >> > > > Mohan >> > > > >> > > > >> > > >> > > -- >> > > -- Guozhang >> > > >> > >> > >> > >> >
