+1 to this FLIP in general. I like the general idea very much (full disclosure, have been involved in the discussions and drafting of the design for a while, so I am not a new/neutral reviewer here).
One thing I would like to see us do here, is reaching out to users early with this, and validating this approach. It is a very fundamental change that also shifts certain tradeoffs, like "cost during execution" vs. "cost on recovery". This approach will increase the data write rate to S3/HDFS/... So before we build every bit of the complex implementation, let's try and validate/test the critical bits with the users. In my assessment, the most critical bit is the continuous log writing, which adds overhead during execution time. Recovery is less critical, there'll be no overhead or additional load, so recovery should be strictly better than currently. I would propose we hence focus on the implementation of the logging first (ignoring recovery, focusing on one target FileSystem/Object store) and test run this with a few users, see that it works well and whether they like the new characteristics. I am also trying to contribute some adjustments to the FLIP text, like more illustrations/explanations, to make it easier to share this FLIP with a wider audience, so we can get the above-mentioned user input and validation. Best, Stephan On Thu, Jan 28, 2021 at 10:46 AM Piotr Nowojski <pnowoj...@apache.org> wrote: > Hi Roman, > > +1 from my side on this proposal. Also big +1 for the recent changes in > this FLIP in the motivation and high level overview sections. > > For me there are quite a bit of unanswered things around how to actually > implement the proposed changes and especially how to integrate it with the > state backends and checkpointing, but maybe we can do that in either a > follow up design docs or discuss it in the tickets or even maybe some PoC. > > Piotrek > > pt., 15 sty 2021 o 07:49 Khachatryan Roman <khachatryan.ro...@gmail.com> > napisał(a): > > > Hi devs, > > > > I'd like to start a discussion of FLIP-158: Generalized incremental > > checkpoints [1] > > > > FLIP motivation: > > Low end-to-end latency is a much-demanded property in many Flink setups. > > With exactly-once, this latency depends on checkpoint interval/duration > > which in turn is defined by the slowest node (usually the one doing a > full > > non-incremental snapshot). In large setups with many nodes, the > probability > > of at least one node being slow gets higher, making almost every > checkpoint > > slow. > > > > This FLIP proposes a mechanism to deal with this by materializing and > > uploading state continuously and only uploading the changed part during > the > > checkpoint itself. It differs from other approaches in that 1) > checkpoints > > are always incremental; 2) works for any state backend. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints > > > > Any feedback highly appreciated! > > > > Regards, > > Roman > > >
