As a follow up to my previous comment, have you thought about writing the checkpoint to a topic instead of a local file? That would have the advantage that all metadata continues to be managed by Kafka, as well as fit with EoS. The potential disadvantage would be a slower latency, however if it is periodic as you mention, I'm not sure that would be a show stopper.
Thanks Eno > On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> wrote: > > Thanks Damian, this is a good idea and will reduce the restore time. Looking > forward, with exactly once and support for transactions in Kafka, I believe > we'll have to add some support for rolling back checkpoints, e.g., when a > transaction is aborted. We need to be aware of that and ideally anticipate a > bit those needs in the KIP. > > Thanks > Eno > > >> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: >> >> Hi all, >> >> I would like to start the discussion on KIP-116: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration >> >> Thanks, >> Damian >
