Thanks for the KIP Damian. I am wondering about two things:
1. what should be the default value for the new parameter? 2. why is the new parameter provided in ms? About (2): because "the minimum checkpoint interval will be the value of commit.interval.ms. In effect the actual checkpoint interval will be a multiple of the commit interval" it might be easier to just use an parameter that is "number-or-commit intervals". -Matthias On 2/1/17 7:29 AM, Damian Guy wrote: > Thanks for the comments Eno. > As for exactly once, i don't believe this matters as we are just restoring > the change-log, i.e, the result of the aggregations that previously ran > etc. So once initialized the state store will be in the same state as it > was before. > Having the checkpoint in a kafka topic is not ideal as the state is per > kafka streams instance. So each instance would need to start with a unique > id that is persistent. > > Cheers, > Damian > > On Wed, 1 Feb 2017 at 13:20 Eno Thereska <eno.there...@gmail.com> wrote: > >> As a follow up to my previous comment, have you thought about writing the >> checkpoint to a topic instead of a local file? That would have the >> advantage that all metadata continues to be managed by Kafka, as well as >> fit with EoS. The potential disadvantage would be a slower latency, however >> if it is periodic as you mention, I'm not sure that would be a show stopper. >> >> Thanks >> Eno >>> On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> wrote: >>> >>> Thanks Damian, this is a good idea and will reduce the restore time. >> Looking forward, with exactly once and support for transactions in Kafka, I >> believe we'll have to add some support for rolling back checkpoints, e.g., >> when a transaction is aborted. We need to be aware of that and ideally >> anticipate a bit those needs in the KIP. >>> >>> Thanks >>> Eno >>> >>> >>>> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: >>>> >>>> Hi all, >>>> >>>> I would like to start the discussion on KIP-116: >>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration >>>> >>>> Thanks, >>>> Damian >>> >> >> >
signature.asc
Description: OpenPGP digital signature
