Re: Nested Iterations Outlook

2015-07-22 Thread Maximilian Alber
Thanks. Yes, I got that. Cheers On Wed, Jul 22, 2015 at 2:46 PM, Maximilian Michels wrote: > I mentioned that. @Max: you should only try it out if you want to > experiment/work with the changes. > > On Wed, Jul 22, 2015 at 2:20 PM, Stephan Ewen wrote: > >> The two pull requests do not go all t

Re: Nested Iterations Outlook

2015-07-22 Thread Maximilian Michels
I mentioned that. @Max: you should only try it out if you want to experiment/work with the changes. On Wed, Jul 22, 2015 at 2:20 PM, Stephan Ewen wrote: > The two pull requests do not go all the way, unfortunately. They cover > only the runtime, the API integration part is missing still, > unfor

Re: Nested Iterations Outlook

2015-07-22 Thread Stephan Ewen
The two pull requests do not go all the way, unfortunately. They cover only the runtime, the API integration part is missing still, unfortunately... On Mon, Jul 20, 2015 at 5:53 PM, Maximilian Michels wrote: > You could do that but you might run into merge conflicts. Also keep in > mind that it

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Michels
You could do that but you might run into merge conflicts. Also keep in mind that it is work in progress :) On Mon, Jul 20, 2015 at 4:15 PM, Maximilian Alber < alber.maximil...@gmail.com> wrote: > Thanks! > > Ok, cool. If I would like to test it, I just need to merge those two pull > requests into

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Alber
Thanks! Ok, cool. If I would like to test it, I just need to merge those two pull requests into my current branch? Cheers, Max On Mon, Jul 20, 2015 at 4:02 PM, Maximilian Michels wrote: > Now that makes more sense :) I thought by "nested iterations" you meant > iterations in Flink that can be

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Michels
Now that makes more sense :) I thought by "nested iterations" you meant iterations in Flink that can be nested, i.e. starting an iteration inside an iteration. The caching/pinning of intermediate results is still a work in progress in Flink. It is actually in a state where it could be merged but s

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Alber
Oh sorry, my fault. When I wrote it, I had iterations in mind. What I actually wanted to say, how "resuming from intermediate results" will work with (non-nested) "non-Flink" iterations? And with iterations I mean something like this: while(...): - change params - submit to cluster where the

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Michels
"So it is up to debate how the support for resuming from intermediate results will look like." -> What's the current state of that debate? Since there is no support for nested iterations that I know of, the debate how intermediate results are integrated has not started yet. > "Intermediate resu

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Alber
Thanks for the answer! But I need some clarification: "So it is up to debate how the support for resuming from intermediate results will look like." -> What's the current state of that debate? "Intermediate results are not produced within the iterations cycles." -> Ok, if there are none, what does

Re: Nested Iterations Outlook

2015-07-20 Thread Maximilian Michels
Hi Max, You are right, there is no support for nested iterations yet. As far as I know, there are no concrete plans to add support for it. So it is up to debate how the support for resuming from intermediate results will look like. Intermediate results are not produced within the iterations cycles

Nested Iterations Outlook

2015-07-17 Thread Maximilian Alber
Hi Flinksters, as far as I know, there is still no support for nested iterations planned. Am I right? So my question is how such use cases should be handled in the future. More specific: when pinning/caching will be available, you suggest to use that feature and program in "Spark" style? Or is th