Big +1 onto this FLIP! Great to see it is stepping forward since this idea is discussed for quite a while. :-)
1. I totally agree that the critical part is the overhead added during normal state updates (forward additional state updates to log as well as state updates itself). Once we have this part ready and assessed, we can have a better understanding/confidence of impact introduced by writing additional log over normal processing. 2. Besides that, it would be also helpful to understand how this idea fits into the overall picture of the fault-tolerance story. But this question might be out of the scope of this FLIP, I guess. Best Yuan On Thu, Jan 28, 2021 at 8:12 PM Stephan Ewen <se...@apache.org> wrote: > +1 to this FLIP in general. > > I like the general idea very much (full disclosure, have been involved in > the discussions and drafting of the design for a while, so I am not a > new/neutral reviewer here). > > One thing I would like to see us do here, is reaching out to users early > with this, and validating this approach. It is a very fundamental change > that also shifts certain tradeoffs, like "cost during execution" vs. "cost > on recovery". This approach will increase the data write rate to > S3/HDFS/... > So before we build every bit of the complex implementation, let's try and > validate/test the critical bits with the users. > > In my assessment, the most critical bit is the continuous log writing, > which adds overhead during execution time. Recovery is less critical, > there'll be no overhead or additional load, so recovery should be strictly > better than currently. > I would propose we hence focus on the implementation of the logging first > (ignoring recovery, focusing on one target FileSystem/Object store) and > test run this with a few users, see that it works well and whether they > like the new characteristics. > > I am also trying to contribute some adjustments to the FLIP text, like more > illustrations/explanations, to make it easier to share this FLIP with a > wider audience, so we can get the above-mentioned user input and > validation. > > Best, > Stephan > > > > > On Thu, Jan 28, 2021 at 10:46 AM Piotr Nowojski <pnowoj...@apache.org> > wrote: > > > Hi Roman, > > > > +1 from my side on this proposal. Also big +1 for the recent changes in > > this FLIP in the motivation and high level overview sections. > > > > For me there are quite a bit of unanswered things around how to actually > > implement the proposed changes and especially how to integrate it with > the > > state backends and checkpointing, but maybe we can do that in either a > > follow up design docs or discuss it in the tickets or even maybe some > PoC. > > > > Piotrek > > > > pt., 15 sty 2021 o 07:49 Khachatryan Roman <khachatryan.ro...@gmail.com> > > napisał(a): > > > > > Hi devs, > > > > > > I'd like to start a discussion of FLIP-158: Generalized incremental > > > checkpoints [1] > > > > > > FLIP motivation: > > > Low end-to-end latency is a much-demanded property in many Flink > setups. > > > With exactly-once, this latency depends on checkpoint interval/duration > > > which in turn is defined by the slowest node (usually the one doing a > > full > > > non-incremental snapshot). In large setups with many nodes, the > > probability > > > of at least one node being slow gets higher, making almost every > > checkpoint > > > slow. > > > > > > This FLIP proposes a mechanism to deal with this by materializing and > > > uploading state continuously and only uploading the changed part during > > the > > > checkpoint itself. It differs from other approaches in that 1) > > checkpoints > > > are always incremental; 2) works for any state backend. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints > > > > > > Any feedback highly appreciated! > > > > > > Regards, > > > Roman > > > > > >
