Hey, it would be great to add this feature indeed! Thanks for bringing it up Greg :) Would the best way be to extend the iteration operators to support intermediate outputs or revisit the idea of caching intermediate results and thus allow efficient for-loop iterations?
-Vasia. On 26 May 2016 at 22:41, Greg Hogan <c...@greghogan.com> wrote: > Hi y'all, > > I think this is an oft-requested feature [0] and there are many graph > algorithms for which intermediate output is the desired result. I'd like to > take Stephan up on his offer [1] for pointers. > > I have yet to get in deep, but I see that iteration tasks are treated > specially as IterationIntermediateTask for synchronization between > supersteps. Also, when OperatorTranslation and GraphCreatingVisitor are > walking the program DAG an iteration must be first reached through the > tail. > > Greg > > [0] > > http://stackoverflow.com/questions/37224140/possibility-of-saving-partial-outputs-from-bulk-iteration-in-flink-dataset > [1] > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Intermediate-output-during-delta-iterations-td436.html >
