Thanks. Yes, I got that. Cheers
On Wed, Jul 22, 2015 at 2:46 PM, Maximilian Michels <m...@apache.org> wrote: > I mentioned that. @Max: you should only try it out if you want to > experiment/work with the changes. > > On Wed, Jul 22, 2015 at 2:20 PM, Stephan Ewen <se...@apache.org> wrote: > >> The two pull requests do not go all the way, unfortunately. They cover >> only the runtime, the API integration part is missing still, >> unfortunately... >> >> On Mon, Jul 20, 2015 at 5:53 PM, Maximilian Michels <m...@apache.org> >> wrote: >> >>> You could do that but you might run into merge conflicts. Also keep in >>> mind that it is work in progress :) >>> >>> On Mon, Jul 20, 2015 at 4:15 PM, Maximilian Alber < >>> alber.maximil...@gmail.com> wrote: >>> >>>> Thanks! >>>> >>>> Ok, cool. If I would like to test it, I just need to merge those two >>>> pull requests into my current branch? >>>> >>>> Cheers, >>>> Max >>>> >>>> On Mon, Jul 20, 2015 at 4:02 PM, Maximilian Michels <m...@apache.org> >>>> wrote: >>>> >>>>> Now that makes more sense :) I thought by "nested iterations" you >>>>> meant iterations in Flink that can be nested, i.e. starting an iteration >>>>> inside an iteration. >>>>> >>>>> The caching/pinning of intermediate results is still a work in >>>>> progress in Flink. It is actually in a state where it could be merged but >>>>> some pending pull requests got delayed because priorities changed a bit. >>>>> >>>>> Essentially, we need to merge these two pull requests: >>>>> >>>>> https://github.com/apache/flink/pull/858 >>>>> This introduces a session management which allows to keep the >>>>> ExecutionGraph for the session. >>>>> >>>>> https://github.com/apache/flink/pull/640 >>>>> Implements the actual backtracking and caching of the results. >>>>> >>>>> Once these are in, we can change the Java/Scala API to support >>>>> backtracking. I don't exactly know how Spark's API does it but, >>>>> essentially >>>>> it should work then by just creating new operations on an existing DataSet >>>>> and submit to the cluster again. >>>>> >>>>> Cheers, >>>>> Max >>>>> >>>>> On Mon, Jul 20, 2015 at 3:31 PM, Maximilian Alber < >>>>> alber.maximil...@gmail.com> wrote: >>>>> >>>>>> Oh sorry, my fault. When I wrote it, I had iterations in mind. >>>>>> >>>>>> What I actually wanted to say, how "resuming from intermediate >>>>>> results" will work with (non-nested) "non-Flink" iterations? And with >>>>>> iterations I mean something like this: >>>>>> >>>>>> while(...): >>>>>> - change params >>>>>> - submit to cluster >>>>>> >>>>>> where the executed Flink-program is more or less the same at each >>>>>> iterations. But with changing input sets, which are reused between >>>>>> different loop iterations. >>>>>> >>>>>> I might got something wrong, because in our group we mentioned >>>>>> caching a lá Spark for Flink and someone came up that "pinning" will do >>>>>> that. Is that somewhat right? >>>>>> >>>>>> Thanks and Cheers, >>>>>> Max >>>>>> >>>>>> On Mon, Jul 20, 2015 at 1:06 PM, Maximilian Michels <m...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> "So it is up to debate how the support for resuming from >>>>>>> intermediate results will look like." -> What's the current state of >>>>>>> that >>>>>>> debate? >>>>>>> >>>>>>> Since there is no support for nested iterations that I know of, the >>>>>>> debate how intermediate results are integrated has not started yet. >>>>>>> >>>>>>> >>>>>>>> "Intermediate results are not produced within the iterations >>>>>>>> cycles." -> Ok, if there are none, what does it have to do with that >>>>>>>> debate? :-) >>>>>>>> >>>>>>> >>>>>>> I was referring to the existing support for intermediate results >>>>>>> within iterations. If we were to implement nested iterations, this could >>>>>>> (possibly) change. This is all very theoretical because there are no >>>>>>> plans >>>>>>> to support nested iterations. >>>>>>> >>>>>>> Hope this clarifies. Otherwise, please restate your question because >>>>>>> I might have misunderstood. >>>>>>> >>>>>>> Cheers, >>>>>>> Max >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 20, 2015 at 12:11 PM, Maximilian Alber < >>>>>>> alber.maximil...@gmail.com> wrote: >>>>>>> >>>>>>>> Thanks for the answer! But I need some clarification: >>>>>>>> >>>>>>>> "So it is up to debate how the support for resuming from >>>>>>>> intermediate results will look like." -> What's the current state of >>>>>>>> that >>>>>>>> debate? >>>>>>>> "Intermediate results are not produced within the iterations >>>>>>>> cycles." -> Ok, if there are none, what does it have to do with that >>>>>>>> debate? :-) >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Max >>>>>>>> >>>>>>>> On Mon, Jul 20, 2015 at 10:50 AM, Maximilian Michels < >>>>>>>> m...@apache.org> wrote: >>>>>>>> >>>>>>>>> Hi Max, >>>>>>>>> >>>>>>>>> You are right, there is no support for nested iterations yet. As >>>>>>>>> far as I know, there are no concrete plans to add support for it. So >>>>>>>>> it is >>>>>>>>> up to debate how the support for resuming from intermediate results >>>>>>>>> will >>>>>>>>> look like. Intermediate results are not produced within the iterations >>>>>>>>> cycles. Same would be true for nested iterations. So the behavior for >>>>>>>>> resuming from intermediate results should be alike for nested >>>>>>>>> iterations. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Max >>>>>>>>> >>>>>>>>> On Fri, Jul 17, 2015 at 4:26 PM, Maximilian Alber < >>>>>>>>> alber.maximil...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Flinksters, >>>>>>>>>> >>>>>>>>>> as far as I know, there is still no support for nested iterations >>>>>>>>>> planned. Am I right? >>>>>>>>>> >>>>>>>>>> So my question is how such use cases should be handled in the >>>>>>>>>> future. >>>>>>>>>> More specific: when pinning/caching will be available, you >>>>>>>>>> suggest to use that feature and program in "Spark" style? Or is >>>>>>>>>> there some >>>>>>>>>> other, more flexible, mechanism planned for loops? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Max >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
