Hi Yun, I did a quick pass over the design doc and it addresses all of the problems with the current iterations I'm aware of. It's great to see that you've been able to workaround the need of vectorized watermarks by giving up nested iterations (which IMO is more of an academic concept than something with a solid use-case). I'll try to give it some more thoughts, but from a first pass it looks great +1 ;)
One thing that I'm unsure about, how do you plan to implement exactly-once checkpointing of the feedback edge? Best, D. On Mon, Oct 4, 2021 at 4:42 AM Yun Gao <yungao...@aliyun.com.invalid> wrote: > Hi all, > > If we do not have other concerns on this FLIP, we would like to start the > voting by the end of oct 8th. > > Best, > Yun. > > > ------------------------------------------------------------------ > From:Yun Gao <yungao...@aliyun.com> > Send Time:2021 Sep. 15 (Wed.) 20:47 > To:dev <dev@flink.apache.org> > Subject:[DISCUSS] FLIP-176: Unified Iteration to Support Algorithms (Flink > ML) > > > Hi all, > > DongLin, ZhiPeng and I are opening this thread to propose designing and > implementing a new iteration library inside the Flink-ML project, as > described in > FLIP-176[1]. > > Iteration serves as a fundamental functionality required to support the > implementation > of ML algorithms. Previously Flink supports bounded iteration on top of > the > DataSet API and unbounded iteration on top of the DataStream API. However, > since we are going to deprecated the dataset API and the current unbounded > iteration > API on top of the DataStream API is not fully complete, thus we are > proposing > to add the new unified iteration library on top of DataStream API to > support both > unbounded and bounded iterations. > > Very thanks for your feedbacks! > > [1] https://cwiki.apache.org/confluence/x/hAEBCw > > Best,Yun
