Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Khachatryan Roman
Thanks a lot for your replies! Yes, feedback is very much appreciated! Especially regarding the approach in general. I think that's a good idea to discuss some details in the follow-up docs or tickets (but I'd be happy to discuss it here as well). As for the PoC, I hope we'll publish it soon (lo

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Yuan Mei
Big +1 onto this FLIP! Great to see it is stepping forward since this idea is discussed for quite a while. :-) 1. I totally agree that the critical part is the overhead added during normal state updates (forward additional state updates to log as well as state updates itself). Once we have this p

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Stephan Ewen
+1 to this FLIP in general. I like the general idea very much (full disclosure, have been involved in the discussions and drafting of the design for a while, so I am not a new/neutral reviewer here). One thing I would like to see us do here, is reaching out to users early with this, and validatin

Re: [DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-28 Thread Piotr Nowojski
Hi Roman, +1 from my side on this proposal. Also big +1 for the recent changes in this FLIP in the motivation and high level overview sections. For me there are quite a bit of unanswered things around how to actually implement the proposed changes and especially how to integrate it with the state

[DISCUSS] FLIP-158: Generalized incremental checkpoints

2021-01-14 Thread Khachatryan Roman
Hi devs, I'd like to start a discussion of FLIP-158: Generalized incremental checkpoints [1] FLIP motivation: Low end-to-end latency is a much-demanded property in many Flink setups. With exactly-once, this latency depends on checkpoint interval/duration which in turn is defined by the slowest no