Another issue with 3rd party state stores could be violation of exactly-once guarantee provided by kafka streams in the event of a failure of streams application instance. Kafka provides exactly once guarantee for consumer -> process -> produce through transactions and with the use of state store like redis, this guarantee is weaker
On Sat, Mar 13, 2021 at 5:28 AM Guozhang Wang <wangg...@gmail.com> wrote: > Hello Mohan, > > I think what you had in mind works with Redis, since it is a remote state > store engine, it does not have the co-partitioning requirements as local > state stores. > > One thing you'd need to tune KS though is that with remote stores, the > processing latency may be larger, and since Kafka Streams process all > records of a single partition in order, synchronously, you may need to tune > the poll interval configs etc to make sure KS would stay in the consumer > group and not trigger unnecessary rebalances. > > Guozhang > > On Thu, Mar 11, 2021 at 6:41 PM Parthasarathy, Mohan <mpart...@hpe.com> > wrote: > > > Hi, > > > > I have a use case where messages come in with some key gets assigned some > > partition and the state gets created. Later, key changes (but still > > contains the old key in the message) and gets sent to a different > > partition. I want to be able to grab the old state using the old key > before > > creating the new state on this instance. Redis as a state store makes it > > easy to implement this where I can simply do a lookup before creating the > > state. I see an implementation here : > > > https://github.com/andreas-schroeder/redisks/tree/master/src/main/java/com/github/andreas_schroeder/redisks > > > > Has anyone tried this ? Any caveats. > > > > Thanks > > Mohan > > > > > > -- > -- Guozhang >
