I'm fine with that. Gouzhang? On Fri, 10 Feb 2017 at 19:45, Matthias J. Sax <matth...@confluent.io> wrote:
> I am actually supporting Eno's view: checkpoint on every commit. > > @Dhwani: I understand your view and did raise the same question about > performance trade-off with checkpoiting enabled/disabled etc. However, > it seems that writing the checkpoint file is super cheap -- thus, there > is nothing to gain performance wise by disabling it. > > For Streams EoS we do not need the checkpoint file -- but we should have > a switch for EoS anyway and can disable the checkpoint file for this > case. And even if there is no switch and we enable EoS all the time, we > can get rid of the checkpoint file overall (making the parameter obsolete). > > IMHO, if the config parameter is not really useful, we should not have it. > > > -Matthias > > > On 2/10/17 9:27 AM, Damian Guy wrote: > > Gouzhang, Thanks for the clarification. Understood. > > > > Eno, you are correct if we just used commit interval then we wouldn't > need > > a KIP. But, then we'd have no way of turning it off. > > > > On Fri, 10 Feb 2017 at 17:14 Eno Thereska <eno.there...@gmail.com> > wrote: > > > >> A quick check: the checkpoint file is not new, we're just exposing a > knob > >> on when to set it, right? Would turning if off still do what it does > today > >> (i.e., write the checkpoint at the end when the user quits?) So it's > not a > >> new feature as such, I was only recommending we dial up the frequency by > >> default. With that option arguably we don't even need a KIP. > >> > >> Eno > >> > >> > >> > >>> On 10 Feb 2017, at 17:02, Guozhang Wang <wangg...@gmail.com> wrote: > >>> > >>> Damian, > >>> > >>> I was thinking if it is a new failure scenarios but as Eno pointed out > it > >>> was not. > >>> > >>> Another thing I was considering is if it has any impact for > incorporating > >>> KIP-98 to avoid duplicates: if there is a failure in the middle of a > >>> transaction, then upon recovery we cannot rely on the local state store > >>> file even if the checkpoint file exists, since the local state store > file > >>> may not be at the transaction boundaries. But since Streams will likely > >> to > >>> have EOS as an opt-in I think it is still worthwhile to add this > feature, > >>> just keeping in mind that when EOS is turned on it may cease to be > >>> effective. > >>> > >>> And yes, I'd suggest we leave the config value to be possibly > >> non-positive > >>> to indicate not turning on this feature for the reason above: if it > will > >>> not be effective then we want to leave it as an option to be turned > off. > >>> > >>> Guozhang > >>> > >>> > >>> On Fri, Feb 10, 2017 at 8:06 AM, Eno Thereska <eno.there...@gmail.com> > >>> wrote: > >>> > >>>> The overhead of writing to the checkpoint file should be much, much > >>>> smaller than the overall overhead of doing a commit, so I think tuning > >> the > >>>> commit time is sufficient to guide performance tradeoffs. > >>>> > >>>> Eno > >>>> > >>>>> On 10 Feb 2017, at 13:08, Dhwani Katagade < > >> dhwani_katag...@persistent.co > >>>> .in> wrote: > >>>>> > >>>>> May be for fine tuning the performance. > >>>>> Say we don't need the checkpointing and would like to gain the lil > bit > >>>> of performance improvement by turning it off. > >>>>> The trade off is between giving people control knobs vs complicating > >> the > >>>> complete set of knobs. > >>>>> > >>>>> -dk > >>>>> > >>>>> On Friday 10 February 2017 04:05 PM, Eno Thereska wrote: > >>>>>> I can't see why users would care to turn it off. > >>>>>> > >>>>>> Eno > >>>>>>> On 10 Feb 2017, at 10:29, Damian Guy <damian....@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi Eno, > >>>>>>> > >>>>>>> Sounds good to me. The only reason i can think of is if we want to > be > >>>> able > >>>>>>> to turn it off. > >>>>>>> Gouzhang - thoughts? > >>>>>>> > >>>>>>> On Fri, 10 Feb 2017 at 10:28 Eno Thereska <eno.there...@gmail.com> > >>>> wrote: > >>>>>>> > >>>>>>>> Question: if checkpointing is so cheap why not do it every commit > >>>>>>>> interval? That way we can get rid of this extra config variable > and > >>>> just > >>>>>>>> use the existing commit interval. > >>>>>>>> > >>>>>>>> Less tuning knobs. > >>>>>>>> > >>>>>>>> Eno > >>>>>>>> > >>>>>>>>> On 10 Feb 2017, at 09:27, Damian Guy <damian....@gmail.com> > wrote: > >>>>>>>>> > >>>>>>>>> Gouzhang, > >>>>>>>>> > >>>>>>>>> You've confused me. The failure scenarios you have described are > >> the > >>>> same > >>>>>>>>> as they are today. With the checkpoint files in place less data > >> will > >>>> be > >>>>>>>>> replayed, so there will be fewer duplicates. > >>>>>>>>> > >>>>>>>>> Are you saying you'd like the option to turn checkpointing off? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Damian > >>>>>>>>> > >>>>>>>>> On Thu, 9 Feb 2017 at 21:55 Guozhang Wang <wangg...@gmail.com> > >>>> wrote: > >>>>>>>>> > >>>>>>>>>> Eno, > >>>>>>>>>> > >>>>>>>>>> You are right, it is not a new scenario. > >>>>>>>>>> > >>>>>>>>>> Thinking a bit more on how we could incorporate KIP-98 in > >> Streams, I > >>>>>>>> feel > >>>>>>>>>> that if EOS is turned on inside Streams, then we probably cannot > >>>> always > >>>>>>>>>> resume from the checkpointed offsets as it is not guaranteed to > be > >>>>>>>>>> "consistent"; but since EOS may not be turned on by default this > >> is > >>>>>>>> still > >>>>>>>>>> worthwhile to add this feature I guess. > >>>>>>>>>> > >>>>>>>>>> About the default config values: I think the default value of 5 > >> min > >>>> is > >>>>>>>> OK > >>>>>>>>>> to me, since restoration is usually faster than normal > processing > >>>>>>>> (unless > >>>>>>>>>> your traffic was really high), about allowing it to be "turned > >> off" > >>>>>>>> with a > >>>>>>>>>> non-positive value: I feel there are still values to keep this > >> door > >>>>>>>> open as > >>>>>>>>>> in the future if EOS is turned on, people may just want to turn > >> off > >>>>>>>>>> checkpointing anyways, or there maybe other scenarios that we > have > >>>> not > >>>>>>>>>> realized yet. On the other hand, I would argue that it is less > >>>> likely > >>>>>>>> users > >>>>>>>>>> mistakenly set it to a non-positive value. > >>>>>>>>>> > >>>>>>>>>> Guozhang > >>>>>>>>>> > >>>>>>>>>> On Thu, Feb 9, 2017 at 1:03 PM, Eno Thereska < > >>>> eno.there...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Guozhang, > >>>>>>>>>>> > >>>>>>>>>>> It seems to me we have the same semantics today. Are you saying > >>>> there > >>>>>>>> is > >>>>>>>>>> a > >>>>>>>>>>> new failure scenario? > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Eno > >>>>>>>>>>> > >>>>>>>>>>>> On 9 Feb 2017, at 19:42, Guozhang Wang <wangg...@gmail.com> > >>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> More specifically, here is my reasoning of failure cases, and > >>>> would > >>>>>>>>>> like > >>>>>>>>>>> to > >>>>>>>>>>>> get your feedbacks: > >>>>>>>>>>>> > >>>>>>>>>>>> *StreamTask* > >>>>>>>>>>>> > >>>>>>>>>>>> For stream-task, the committing order is 1) flush state (may > >> send > >>>> more > >>>>>>>>>>>> records to changelog in producer), 2) flush producer, 3) > commit > >>>>>>>>>> upstream > >>>>>>>>>>>> offsets. My understanding is that the writing of the > checkpoint > >>>> file > >>>>>>>>>> will > >>>>>>>>>>>> between 2) and 3). So thatt he new order will be 1) flush > state, > >>>> 2) > >>>>>>>>>> flush > >>>>>>>>>>>> producer, 3) write checkpoint file (when necessary), 4) commit > >>>>>>>> upstream > >>>>>>>>>>>> offsets. > >>>>>>>>>>>> > >>>>>>>>>>>> And we have a bunch of "changelog offsets" regarding the > state: > >> a) > >>>>>>>>>> offset > >>>>>>>>>>>> corresponding to the image of the persistent file, name it > point > >>>> A, b) > >>>>>>>>>>> log > >>>>>>>>>>>> end offset, name it offset B, c) checkpoint file recorded > >> offset, > >>>> name > >>>>>>>>>> it > >>>>>>>>>>>> offset C, d) offset corresponding to the current committed > >>>> upstream > >>>>>>>>>>> offset, > >>>>>>>>>>>> name it offset D. > >>>>>>>>>>>> > >>>>>>>>>>>> Now let's talk about the failure cases: > >>>>>>>>>>>> > >>>>>>>>>>>> If there is a crash between 1) and 2), then A > B = C = D. In > >> this > >>>>>>>>>> case, > >>>>>>>>>>> if > >>>>>>>>>>>> we restore, we will replay no logs at all since B = C while > the > >>>>>>>>>>> persistent > >>>>>>>>>>>> state file is actually "ahead of time", and we will start > >>>> reprocessing > >>>>>>>>>>>> since from the input offset corresponding to D = B < A and > hence > >>>> have > >>>>>>>>>>> some > >>>>>>>>>>>> duplicated, *which will be incorrect* if the update logic > >> involve > >>>>>>>>>> reading > >>>>>>>>>>>> the state store values as well (i.e. not a blind write), e.g. > >>>>>>>>>>> aggregations. > >>>>>>>>>>>> If there is a crash between 2) and 3), then A = B > C = D. > When > >> we > >>>>>>>>>>> restore, > >>>>>>>>>>>> we will replay from C -> B = A, and then start reprocessing > from > >>>> input > >>>>>>>>>>>> offset corresponding to D < A, and same issue applies as > above. > >>>>>>>>>>>> > >>>>>>>>>>>> If there is a crash between 3) and 4), then A = B = C > D. > When > >> we > >>>>>>>>>>> restore, > >>>>>>>>>>>> we will not replay, and then start reprocessing from input > >> offset > >>>>>>>>>>>> corresponding to D < A, and same issue applies as above. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> *StandbyTask* > >>>>>>>>>>>> > >>>>>>>>>>>> We only do one operation today, which is 1) flush state, I > think > >>>> we > >>>>>>>>>> will > >>>>>>>>>>>> add the writing of the checkpoint file after it as step 2). > >>>>>>>>>>>> > >>>>>>>>>>>> Failure cases again: offset A -> correspond to the image of > the > >>>> file, > >>>>>>>>>>>> offset B -> changelog end offset, offset C -> written as in > the > >>>>>>>>>>> checkpoint > >>>>>>>>>>>> file. > >>>>>>>>>>>> > >>>>>>>>>>>> If there is a crash between 1) and 2), then B >= A > C (B >= A > >>>> because > >>>>>>>>>> we > >>>>>>>>>>>> are reading from changelog topic so A will never be greater > than > >>>> B), > >>>>>>>>>>>> > >>>>>>>>>>>> 1) and if this task resumes as a standby task, we will resume > >>>>>>>>>> restoration > >>>>>>>>>>>> from offset C, and a few duplicates from C -> A will be > applied > >>>> again > >>>>>>>>>> to > >>>>>>>>>>>> local state files, then continue from A -> B, *this is OK* > since > >>>> they > >>>>>>>>>> do > >>>>>>>>>>>> not incur any computations hence no side effects and are all > >>>>>>>>>> idempotent. > >>>>>>>>>>>> 2) and if this task resumes as a stream task, we will replay > >>>>>>>> changelogs > >>>>>>>>>>>> from C -> A, with duplicated updates, and then from A -> B. > This > >>>> is > >>>>>>>>>> also > >>>>>>>>>>> OK > >>>>>>>>>>>> for the same reason as above. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> So it seems to me that this is not safe for a StreamTask, or > >>>> maybe the > >>>>>>>>>>>> writing of the checkpoint file in your mind is different? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Guozhang > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Feb 9, 2017 at 11:02 AM, Guozhang Wang < > >>>> wangg...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> A quick question re: `We will add the above config parameter > to > >>>>>>>>>>>>> *StreamsConfig*. During *StreamTask#commit()*, > >>>>>>>> *StandbyTask#commit()*, > >>>>>>>>>>>>> and *GlobalUpdateStateTask#flushState()* we will check if the > >>>>>>>>>>> checkpoint > >>>>>>>>>>>>> interval has elapsed and write the checkpoint file.` > >>>>>>>>>>>>> > >>>>>>>>>>>>> Will the writing of the checkpoint file happen before the > >>>> flushing of > >>>>>>>>>>> the > >>>>>>>>>>>>> state manager? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Guozhang > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Feb 9, 2017 at 10:48 AM, Matthias J. Sax < > >>>>>>>>>> matth...@confluent.io > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> But 5 min means, that we (in the worst case) need to reply > >> data > >>>> from > >>>>>>>>>>> the > >>>>>>>>>>>>>> last 5 minutes to get the store ready. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> So why not go with the min possible value of 30 seconds to > >>>> speed up > >>>>>>>>>>> this > >>>>>>>>>>>>>> process if the impact is negligible anyway? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> What do you gain by being conservative? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/9/17 2:54 AM, Damian Guy wrote: > >>>>>>>>>>>>>>> Why shouldn't it be 5 minutes? ;-) > >>>>>>>>>>>>>>> It is a finger in the air number. Based on the testing i > did > >> it > >>>>>>>>>> shows > >>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> there isn't much, if any, overhead when checkpointing a > >> single > >>>>>>>> store > >>>>>>>>>>> on > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> commit interval. The default commit interval is 30 seconds, > >> so > >>>> it > >>>>>>>>>>> could > >>>>>>>>>>>>>>> possibly be set to that. However, i'd prefer to be a little > >>>>>>>>>>>>>> conservative so > >>>>>>>>>>>>>>> 5 minutes seemed reasonable. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, 9 Feb 2017 at 10:25 Michael Noll < > >> mich...@confluent.io > >>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> Damian, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> could you elaborate briefly why the default value should > be > >> 5 > >>>>>>>>>>> minutes? > >>>>>>>>>>>>>>>> What are the considerations, assumptions, etc. that go > into > >>>>>>>> picking > >>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>> value? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Right now, in the KIP and in this discussion, "5 mins" > looks > >>>> like > >>>>>>>> a > >>>>>>>>>>>>>> magic > >>>>>>>>>>>>>>>> number to me. :-) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -Michael > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Thu, Feb 9, 2017 at 11:03 AM, Damian Guy < > >>>> damian....@gmail.com > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> I've ran the SimpleBenchmark with checkpoint on and off > to > >>>> see > >>>>>>>>>> what > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> impact is. It appears that there is very little impact, > if > >>>> any. > >>>>>>>>>> The > >>>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>>>> with checkpointing on actually look better, but that is > >>>> likely > >>>>>>>>>>> largely > >>>>>>>>>>>>>>>> due > >>>>>>>>>>>>>>>>> to external influences. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> In any case, i'm going to suggest we go with a default > >>>> checkpoint > >>>>>>>>>>>>>>>> interval > >>>>>>>>>>>>>>>>> of 5 minutes. I've update the KIP with this. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> commit every 10 seconds (no checkpoint) > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/34798/287372.83751939767/29.570664980746017 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/35942/278226.0308274442/28.62945857214401 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/34677/288375.58035585546/29.673847218617528 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/34677/288375.58035585546/29.673847218617528 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/31192/320595.02436522185/32.98922800718133 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> checkpoint every 10 seconds (same as commit interval) > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/36997/270292.185852907/27.81306592426413 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/32087/311652.69423754164/32.069062237043035 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/32895/303997.5680194558/31.281349749202004 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/33476/298721.4720994145/30.738439479029754 > >>>>>>>>>>>>>>>>> Streams Performance [records/latency/rec-sec/MB-sec > >>>>>>>> source+store]: > >>>>>>>>>>>>>>>>> 10000000/33196/301241.1133871551/30.99771056753826 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, 8 Feb 2017 at 09:02 Damian Guy < > >> damian....@gmail.com > >>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Matthias, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Fair point. I'll update it the KIP. > >>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Wed, 8 Feb 2017 at 05:49 Matthias J. Sax < > >>>>>>>>>> matth...@confluent.io > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> Damian, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I am not strict about it either. However, if there is no > >>>>>>>>>> advantage > >>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> disabling it, we might not want to allow it. This would > >>>> have the > >>>>>>>>>>>>>>>>>> advantage to guard users to accidentally switch it off. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/3/17 2:03 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>> Hi Matthias, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> It possibly doesn't make sense to disable it, but then > >> i'm > >>>> sure > >>>>>>>>>>>>>>>> someone > >>>>>>>>>>>>>>>>>>> will come up with a reason they don't want it! > >>>>>>>>>>>>>>>>>>> I'm happy to change it such that the checkpoint > interval > >>>> must > >>>>>>>>>> be > > >>>>>>>>>>>>>> 0. > >>>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>>> Damian > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Fri, 3 Feb 2017 at 01:29 Matthias J. Sax < > >>>>>>>>>>> matth...@confluent.io> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> Thanks Damian. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> One more question: "Checkpointing is disabled if the > >>>>>>>> checkpoint > >>>>>>>>>>>>>>>>> interval > >>>>>>>>>>>>>>>>>>>> is set to a value <=0." > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Does it make sense to disable check pointing? What's > the > >>>>>>>>>> tradeoff > >>>>>>>>>>>>>>>>> here? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 2/2/17 1:51 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>> Hi Matthias, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> 1. TBD - i need to do some performance tests and try > >> and > >>>> work > >>>>>>>>>>> out > >>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>> sensible default. > >>>>>>>>>>>>>>>>>>>>> 2. Yes, you are correct. It could be a multiple of > the > >>>>>>>>>>>>>>>>>>>> commit.interval.ms. > >>>>>>>>>>>>>>>>>>>>> But, that would also mean if you change the commit > >>>> interval - > >>>>>>>>>>> say > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>> lower > >>>>>>>>>>>>>>>>>>>>> it, then you might also need to change the checkpoint > >>>> setting > >>>>>>>>>>>>>> (i.e, > >>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>> still only want to checkpoint every n minutes). > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Wed, 1 Feb 2017 at 23:46 Matthias J. Sax < > >>>>>>>>>>>>>> matth...@confluent.io > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Damian. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I am wondering about two things: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 1. what should be the default value for the new > >>>> parameter? > >>>>>>>>>>>>>>>>>>>>>> 2. why is the new parameter provided in ms? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> About (2): because > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> "the minimum checkpoint interval will be the value > of > >>>>>>>>>>>>>>>>>>>>>> commit.interval.ms. In effect the actual checkpoint > >>>>>>>> interval > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>> multiple of the commit interval" > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> it might be easier to just use an parameter that is > >>>>>>>>>>>>>>>>> "number-or-commit > >>>>>>>>>>>>>>>>>>>>>> intervals". > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 2/1/17 7:29 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the comments Eno. > >>>>>>>>>>>>>>>>>>>>>>> As for exactly once, i don't believe this matters > as > >>>> we are > >>>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>>>> restoring > >>>>>>>>>>>>>>>>>>>>>>> the change-log, i.e, the result of the aggregations > >>>> that > >>>>>>>>>>>>>>>> previously > >>>>>>>>>>>>>>>>>> ran > >>>>>>>>>>>>>>>>>>>>>>> etc. So once initialized the state store will be in > >> the > >>>>>>>> same > >>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>> was before. > >>>>>>>>>>>>>>>>>>>>>>> Having the checkpoint in a kafka topic is not ideal > >> as > >>>> the > >>>>>>>>>>> state > >>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> per > >>>>>>>>>>>>>>>>>>>>>>> kafka streams instance. So each instance would need > >> to > >>>>>>>> start > >>>>>>>>>>>>>>>> with a > >>>>>>>>>>>>>>>>>>>>>> unique > >>>>>>>>>>>>>>>>>>>>>>> id that is persistent. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>>>>>>> Damian > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Wed, 1 Feb 2017 at 13:20 Eno Thereska < > >>>>>>>>>>>>>> eno.there...@gmail.com > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> As a follow up to my previous comment, have you > >>>> thought > >>>>>>>>>> about > >>>>>>>>>>>>>>>>>> writing > >>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> checkpoint to a topic instead of a local file? > That > >>>> would > >>>>>>>>>>> have > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> advantage that all metadata continues to be > managed > >> by > >>>>>>>>>> Kafka, > >>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>> well > >>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>> fit with EoS. The potential disadvantage would be > a > >>>> slower > >>>>>>>>>>>>>>>>> latency, > >>>>>>>>>>>>>>>>>>>>>> however > >>>>>>>>>>>>>>>>>>>>>>>> if it is periodic as you mention, I'm not sure > that > >>>> would > >>>>>>>>>> be > >>>>>>>>>>> a > >>>>>>>>>>>>>>>>> show > >>>>>>>>>>>>>>>>>>>>>> stopper. > >>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>>>>> On 1 Feb 2017, at 12:58, Eno Thereska < > >>>>>>>>>>> eno.there...@gmail.com > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks Damian, this is a good idea and will > reduce > >>>> the > >>>>>>>>>>> restore > >>>>>>>>>>>>>>>>>> time. > >>>>>>>>>>>>>>>>>>>>>>>> Looking forward, with exactly once and support for > >>>>>>>>>>> transactions > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>> Kafka, I > >>>>>>>>>>>>>>>>>>>>>>>> believe we'll have to add some support for rolling > >>>> back > >>>>>>>>>>>>>>>>> checkpoints, > >>>>>>>>>>>>>>>>>>>>>> e.g., > >>>>>>>>>>>>>>>>>>>>>>>> when a transaction is aborted. We need to be aware > >> of > >>>> that > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> ideally > >>>>>>>>>>>>>>>>>>>>>>>> anticipate a bit those needs in the KIP. > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 1 Feb 2017, at 10:18, Damian Guy < > >>>>>>>>>> damian....@gmail.com> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start the discussion on KIP-116: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>>>>>>> 116+-+Add+State+Store+Checkpoint+Interval+Configuration > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>> Damian > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> -- Guozhang > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> -- Guozhang > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> -- Guozhang > >>>>>>>>>> > >>>>>>>> > >>>>> > >>>>> > >>>>> DISCLAIMER > >>>>> ========== > >>>>> This e-mail may contain privileged and confidential information which > >> is > >>>> the property of Persistent Systems Ltd. It is intended only for the > use > >> of > >>>> the individual or entity to which it is addressed. If you are not the > >>>> intended recipient, you are not authorized to read, retain, copy, > print, > >>>> distribute or use this message. If you have received this > communication > >> in > >>>> error, please notify the sender and delete all copies of this message. > >>>> Persistent Systems Ltd. does not accept any liability for virus > infected > >>>> mails. > >>>>> > >>>> > >>>> > >>> > >>> > >>> -- > >>> -- Guozhang > >> > >> > > > >
