It depends on the store implementation. Atm, EOS for state store is achieved by re-creating the state store in case of failure from the changelog topic.
For RocksDB stores, we wipe out the local state directories and create a new empty RocksDB and for in-memory stores the content is "lost" anyway when state is migrated, and we reply the changelog into an empty store before processing resumes. -Matthias On 1/5/21 6:27 AM, Alex Craig wrote: > I don't think he's asking about data-loss, but rather data consistency. > (in the event of an exception or application crash, will EOS ensure that > the state store data is consistent) My understanding is that it DOES apply > to state stores as well, in the sense that a failure during processing > would mean that the commit wouldn't get flushed and therefore wouldn't get > double-counted once processing resumes and message is re-processed. > As far as using something other than RocksDB, I think as long as you are > implementing the state store API correctly you should be fine. I did a POC > recently using Mongo state-stores with EOS enabled and it worked correctly, > even when I intentionally introduced failures and crashes. > > -alex > > On Tue, Jan 5, 2021 at 1:09 AM Ning Zhang <ning2008w...@gmail.com> wrote: > >> If there is a "change-log" topic to back up the state store, then it may >> not lose data. >> >> Also, if the third party store is not "kafka community certified" (or not >> well-maintained), it may have chances to lose data (in different ways). >> >> >> >> On 2021/01/05 04:56:12, Pushkar Deole <pdeole2...@gmail.com> wrote: >>> In case we opt to choose some third party store instead of kafka's stores >>> for storing state (e.g. Redis cache or Ignite), then will we lose the >>> exactly-once guarantee provided by kafka and the state stores can be in >> an >>> inconsistent state ? >>> >>> On Sat, Jan 2, 2021 at 4:56 AM Ning Zhang <ning2008w...@gmail.com> >> wrote: >>> >>>> The physical store behind "state store" is change-log kafka topic. In >>>> Kafka stream, if something fails in the middle, the "state store" is >>>> restored back to the state before the event happens at the first step / >>>> beginning of the stream. >>>> >>>> >>>> >>>> On 2020/12/31 08:48:16, Pushkar Deole <pdeole2...@gmail.com> wrote: >>>>> Hi All, >>>>> >>>>> We use Kafka streams and may need to use exactly-once configuration >> for >>>>> some of the use cases. Currently, the application uses either local >> or >>>>> global state store to store state. >>>>> So, the application will consume events from source kafka topic, >> process >>>>> the events, for state stores it will use either local or global state >>>> store >>>>> of kafka, then produce events onto the destination topic. >>>>> >>>>> Question i have is: in the case of exactly-once setting, kafka >> streams >>>>> guarantees that all actions happen or nothing happens. So, in this >> case, >>>>> any state stored on the local or global state store will also be >> counted >>>>> under 'all or nothing' guarantee e.g. if event is consumed and state >>>> store >>>>> is updated, however some issue occurs before event is produced on >>>>> destination topic then will state store be restored back to the state >>>>> before it was updated for this event? >>>>> >>>> >>> >> >
