If there is a "change-log" topic to back up the state store, then it may not lose data.
Also, if the third party store is not "kafka community certified" (or not well-maintained), it may have chances to lose data (in different ways). On 2021/01/05 04:56:12, Pushkar Deole <pdeole2...@gmail.com> wrote: > In case we opt to choose some third party store instead of kafka's stores > for storing state (e.g. Redis cache or Ignite), then will we lose the > exactly-once guarantee provided by kafka and the state stores can be in an > inconsistent state ? > > On Sat, Jan 2, 2021 at 4:56 AM Ning Zhang <ning2008w...@gmail.com> wrote: > > > The physical store behind "state store" is change-log kafka topic. In > > Kafka stream, if something fails in the middle, the "state store" is > > restored back to the state before the event happens at the first step / > > beginning of the stream. > > > > > > > > On 2020/12/31 08:48:16, Pushkar Deole <pdeole2...@gmail.com> wrote: > > > Hi All, > > > > > > We use Kafka streams and may need to use exactly-once configuration for > > > some of the use cases. Currently, the application uses either local or > > > global state store to store state. > > > So, the application will consume events from source kafka topic, process > > > the events, for state stores it will use either local or global state > > store > > > of kafka, then produce events onto the destination topic. > > > > > > Question i have is: in the case of exactly-once setting, kafka streams > > > guarantees that all actions happen or nothing happens. So, in this case, > > > any state stored on the local or global state store will also be counted > > > under 'all or nothing' guarantee e.g. if event is consumed and state > > store > > > is updated, however some issue occurs before event is produced on > > > destination topic then will state store be restored back to the state > > > before it was updated for this event? > > > > > >
