Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-16 Thread Arvid Heise
Voting period is now over even with the roadmap changes (forgot to close on Friday because of all the Coronavirus chaos). We have 4 binding votes (Thomas, Yu, Piotr, Zhijiang) and no objections, so FLIP-76 passed. Thank you very much for your feedback. On Fri, Mar 13, 2020 at 11:08 AM Yu Li wro

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-13 Thread Yu Li
+1 (binding) The updated FLIP doc LGTM. Thanks for addressing the comments Arvid and Roman. Best Regards, Yu On Fri, 13 Mar 2020 at 03:48, Arvid Heise wrote: > I added a roadmap section to the FLIP as suggested by Yu and Roman. > > Unless someone objects, I'd still consider the voting period

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Yingjie Cao
+1 (non-binding) Checkpoint timeout in cases of backpressure is hard to tune. I and our users ever spent lots of time on that. It is great to have this feature. Arvid Heise 于2020年3月10日周二 下午9:33写道: > Hi all, > > I would like to start the vote for FLIP-76 [1], which is discussed and > reached a c

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Arvid Heise
I added a roadmap section to the FLIP as suggested by Yu and Roman. Unless someone objects, I'd still consider the voting period to end tomorrow. For me, the roadmap is only a clarification of already written and discussed points. We already have enough binding votes, but there may be concerns po

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-12 Thread Yun Gao
+1 (non-binding) I think the PoC result has shown the effect on reducing checkpoint time when back-pressure occurs, and I totally agree with that the feature could be implemented in steps. -- From:Roman Khachatryan Send Time:

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Roman Khachatryan
+1 (non-binding) Regarding Yu's suggestion about *Roadmap* or *Future Work* section, I think it's a good idea. Currently, some MVP limitations are mentioned at the end of the document, so we can extract and expand it. As for the recovery speed it's not a priority currently, but we could also menti

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Zhijiang
+1 (binding). As for David's concern of smaller buffers after recovery, I ever had a draft design [1] to solve this issue. You can take a look and leave comments if still have concerns. :) [1] https://docs.google.com/document/d/16_MOQymzxrKvUHXh6QFr2AAXIKt_2vPUf8vzKy4H_tU/edit Best, Zhijiang

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Piotr Nowojski
+1 (binding). Piotrek > On 11 Mar 2020, at 09:19, David Anderson wrote: > > +1 I like where this is headed. > > One question: during restore, it could happen that a new task manager is > configured with fewer or smaller buffers than was previously the case. How > will this be handled? > > Dav

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread David Anderson
+1 I like where this is headed. One question: during restore, it could happen that a new task manager is configured with fewer or smaller buffers than was previously the case. How will this be handled? David On Wed, Mar 11, 2020 at 8:31 AM Arvid Heise wrote: > Hi Thomas, > > it's like you sai

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-11 Thread Arvid Heise
Hi Thomas, it's like you said. The first version will not support rescaling and mostly addresses the concerns about making little to no progress because of frequent crashes. The main reason is that we cannot guarantee the ordering of non-keyed data (and even keyed data in some weird cases) when r

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-10 Thread Yu Li
+1 on the overall design and thanks for the efforts! I totally agree with the plan of implementing the MVP first. However, since the FLIP is for the whole feature instead of only MVP, how about adding a *Roadmap* or *Future Work* section to write down plans include (but not limited to): * Dynamic

Re: [VOTE] [FLIP-76] Unaligned checkpoints

2020-03-10 Thread Thomas Weise
+1 Thanks for putting this together, looking forward to the experimental support in the next release. One clarification: since the MVP won't support rescaling, does it imply that savepoints will always use aligned checkpointing? If so, this would still block the user from taking a savepoint and r