Re: Iteration Intermediate Output

2016-05-30 Thread Andrew Palumbo
ut/flinkbindings/drm/CheckpointedFlinkDrm.scala#L139 From: Greg Hogan Sent: Thursday, May 26, 2016 4:41:53 PM To: dev@flink.apache.org Subject: Iteration Intermediate Output Hi y'all, I think this is an oft-requested feature [0] and there are many graph algorithms for

Re: Iteration Intermediate Output

2016-05-30 Thread Kostas Tzoumas
Thanks Greg for opening this discussion! I really really don't want to derail the discussion here, just a quick clarification regarding Suneel's last email: folks that are working at data Artisans are participating in this community as individuals, not as a corporation, and the dev list is not a s

Re: Iteration Intermediate Output

2016-05-30 Thread Suneel Marthi
This is a feature that was requested by the Mahout project few months before for the very same reasons as mentioned in previous emails on this thread, but we were snubbed by the flink folks as this being '*WAY too specific*' request for flink to deal with and 'its got to be done the way Flink has i

Re: Iteration Intermediate Output

2016-05-30 Thread Gábor Gévay
Hello, > Would the best way be to extend the iteration operators to support > intermediate outputs or revisit the idea of caching intermediate results > and thus allow efficient for-loop iterations? Caching intermediate results would also help a lot to projects that are targeting Flink as a backe

Re: Iteration Intermediate Output

2016-05-28 Thread Vasiliki Kalavri
Hey, it would be great to add this feature indeed! Thanks for bringing it up Greg :) Would the best way be to extend the iteration operators to support intermediate outputs or revisit the idea of caching intermediate results and thus allow efficient for-loop iterations? -Vasia. On 26 May 2016 at

Iteration Intermediate Output

2016-05-26 Thread Greg Hogan
Hi y'all, I think this is an oft-requested feature [0] and there are many graph algorithms for which intermediate output is the desired result. I'd like to take Stephan up on his offer [1] for pointers. I have yet to get in deep, but I see that iteration tasks are treated specially as IterationIn