Thanks for the comments Eno. As for exactly once, i don't believe this matters as we are just restoring the change-log, i.e, the result of the aggregations that previously ran etc. So once initialized the state store will be in the same state as it was before. Having the checkpoint in a kafka topic is not ideal as the state is per kafka streams instance. So each instance would need to start with a unique id that is persistent.
Cheers, Damian On Wed, 1 Feb 2017 at 13:20 Eno Thereska <eno.there...@gmail.com> wrote: > As a follow up to my previous comment, have you thought about writing the > checkpoint to a topic instead of a local file? That would have the > advantage that all metadata continues to be managed by Kafka, as well as > fit with EoS. The potential disadvantage would be a slower latency, however > if it is periodic as you mention, I'm not sure that would be a show stopper. > > Thanks > Eno > > On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> wrote: > > > > Thanks Damian, this is a good idea and will reduce the restore time. > Looking forward, with exactly once and support for transactions in Kafka, I > believe we'll have to add some support for rolling back checkpoints, e.g., > when a transaction is aborted. We need to be aware of that and ideally > anticipate a bit those needs in the KIP. > > > > Thanks > > Eno > > > > > >> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: > >> > >> Hi all, > >> > >> I would like to start the discussion on KIP-116: > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration > >> > >> Thanks, > >> Damian > > > >
