I don't think he's asking about data-loss, but rather data consistency. (in the event of an exception or application crash, will EOS ensure that the state store data is consistent) My understanding is that it DOES apply to state stores as well, in the sense that a failure during processing would mean that the commit wouldn't get flushed and therefore wouldn't get double-counted once processing resumes and message is re-processed. As far as using something other than RocksDB, I think as long as you are implementing the state store API correctly you should be fine. I did a POC recently using Mongo state-stores with EOS enabled and it worked correctly, even when I intentionally introduced failures and crashes.
-alex On Tue, Jan 5, 2021 at 1:09 AM Ning Zhang <ning2008w...@gmail.com> wrote: > If there is a "change-log" topic to back up the state store, then it may > not lose data. > > Also, if the third party store is not "kafka community certified" (or not > well-maintained), it may have chances to lose data (in different ways). > > > > On 2021/01/05 04:56:12, Pushkar Deole <pdeole2...@gmail.com> wrote: > > In case we opt to choose some third party store instead of kafka's stores > > for storing state (e.g. Redis cache or Ignite), then will we lose the > > exactly-once guarantee provided by kafka and the state stores can be in > an > > inconsistent state ? > > > > On Sat, Jan 2, 2021 at 4:56 AM Ning Zhang <ning2008w...@gmail.com> > wrote: > > > > > The physical store behind "state store" is change-log kafka topic. In > > > Kafka stream, if something fails in the middle, the "state store" is > > > restored back to the state before the event happens at the first step / > > > beginning of the stream. > > > > > > > > > > > > On 2020/12/31 08:48:16, Pushkar Deole <pdeole2...@gmail.com> wrote: > > > > Hi All, > > > > > > > > We use Kafka streams and may need to use exactly-once configuration > for > > > > some of the use cases. Currently, the application uses either local > or > > > > global state store to store state. > > > > So, the application will consume events from source kafka topic, > process > > > > the events, for state stores it will use either local or global state > > > store > > > > of kafka, then produce events onto the destination topic. > > > > > > > > Question i have is: in the case of exactly-once setting, kafka > streams > > > > guarantees that all actions happen or nothing happens. So, in this > case, > > > > any state stored on the local or global state store will also be > counted > > > > under 'all or nothing' guarantee e.g. if event is consumed and state > > > store > > > > is updated, however some issue occurs before event is produced on > > > > destination topic then will state store be restored back to the state > > > > before it was updated for this event? > > > > > > > > > >
