This is what I understand could be the issue with external state store: kafka stream application consumes source topic, does processing, stores state to kafka state store (this is backed by topic) and before producing event on destination topic, the application fails with some issue. In this case, the transaction has failed, so kafka guarantees either all or none, means offset written to source topic, state written to state store topic, output produced on destination topic... all of these happen or none of these and in this failure scenario it is none of these.
Assume you have redis state store, and you updated the state into redis and stream application failed. Now, you have source topic and destination topic consistent i.e. offset is not committed to source topic and output not produced on destination topic, but you redis state store is inconsistent with that since it is external state store and kafka can't guarantee rollback ot state written there On Mon, Mar 15, 2021 at 6:30 PM Alex Craig <alexcrai...@gmail.com> wrote: > " Another issue with 3rd party state stores could be violation of > exactly-once guarantee provided by kafka streams in the event of a failure > of streams application instance" > > I've heard this before but would love to know more about how a custom state > store would be at any greater risk than RocksDB as far as exactly-once > guarantees are concerned. They all implement the same interface, so as > long as you're correctly implementing get(), put(), delete(), flush(), etc, > you should be fine right? In other words, I don't think there is any > special "exactly once magic" that is baked into the RocksDB store code. I > could be wrong though so I'd love to hear people's thoughts, thanks, > > Alex C > > On Sun, Mar 14, 2021 at 4:58 PM Parthasarathy, Mohan <mpart...@hpe.com> > wrote: > > > Thanks for the responses. In the worst case, I might have to keep both > > rocksdb for local store and keep an external store like Redis. > > > > -mohan > > > > > > On 3/13/21, 8:53 PM, "Pushkar Deole" <pdeole2...@gmail.com> wrote: > > > > Another issue with 3rd party state stores could be violation of > > exactly-once guarantee provided by kafka streams in the event of a > > failure > > of streams application instance. > > Kafka provides exactly once guarantee for consumer -> process -> > > produce > > through transactions and with the use of state store like redis, this > > guarantee is weaker > > > > On Sat, Mar 13, 2021 at 5:28 AM Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > Hello Mohan, > > > > > > I think what you had in mind works with Redis, since it is a remote > > state > > > store engine, it does not have the co-partitioning requirements as > > local > > > state stores. > > > > > > One thing you'd need to tune KS though is that with remote stores, > > the > > > processing latency may be larger, and since Kafka Streams process > all > > > records of a single partition in order, synchronously, you may need > > to tune > > > the poll interval configs etc to make sure KS would stay in the > > consumer > > > group and not trigger unnecessary rebalances. > > > > > > Guozhang > > > > > > On Thu, Mar 11, 2021 at 6:41 PM Parthasarathy, Mohan < > > mpart...@hpe.com> > > > wrote: > > > > > > > Hi, > > > > > > > > I have a use case where messages come in with some key gets > > assigned some > > > > partition and the state gets created. Later, key changes (but > still > > > > contains the old key in the message) and gets sent to a different > > > > partition. I want to be able to grab the old state using the old > > key > > > before > > > > creating the new state on this instance. Redis as a state store > > makes it > > > > easy to implement this where I can simply do a lookup before > > creating the > > > > state. I see an implementation here : > > > > > > > > > > https://github.com/andreas-schroeder/redisks/tree/master/src/main/java/com/github/andreas_schroeder/redisks > > > > > > > > Has anyone tried this ? Any caveats. > > > > > > > > Thanks > > > > Mohan > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > >
