Thanks Gabor. I'll keep an eye on the developments. Supun..
On Thu, Sep 1, 2016 at 12:57 PM, Gábor Gévay <gga...@gmail.com> wrote: > I don't think that there are plans for enabling the nesting of the > native iteration constructs, but we should wait for one of the > commiters to confirm this. > > However, the matter of caching of intermediate results has came up on > numerous occasions before [1,2,3,4,5], and it would be useful in lots > of other situations as well, so there is hope that it will be > implemented some day, which would make the 1. workaround from above > more feasible. > > Best, > Gábor > > [1] https://issues.apache.org/jira/browse/FLINK-1730 > [2] http://apache-flink-mailing-list-archive.1008284.n3. > nabble.com/Iteration-Intermediate-Output-td11850.html > [3] http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Questions-re-ExecutionGraph-amp-ResultPartitions-for- > interactive-use-a-la-Spark-td4154.html > [4] http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Flink-programm-with-for-loop-yields-wrong- > results-when-run-in-parallel-td7783.html > [5] http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Iterative-queries-on-Flink-td3786.html > > > > > 2016-09-01 18:31 GMT+02:00 Supun Kamburugamuve <supu...@gmail.com>: > > Thanks Gabor. I was thinking about starting separate jobs. > > > > Is there any plans to support nested loops in the future? > > > > Thanks, > > Supun.. > > > > On Thu, Sep 1, 2016 at 12:28 PM, Gábor Gévay <gga...@gmail.com> wrote: > >> > >> Hello Supun, > >> > >> Unfortunately, nesting of Flink's iteration constructs are not > >> supported at the moment. > >> > >> There are some workarounds though: > >> > >> 1. You can start a Flink job for each step of the iteration. Starting > >> a Flink job has some overhead, so this only works if there is a > >> sufficient amount of work in each iteration step. Moreover, this has > >> the disadvantage that the intermediate results are always need to be > >> written out and then read back between steps, which might have a > >> considerable performance impact. > >> > >> 2. If you have just a small fixed number of steps, then you can have a > >> for loop that "unrolls" all the iteration steps, and creates one large > >> Flink job. The code will be somewhat similar to the first approach, > >> but you don't call execute between the steps, and you don't write > >> intermediate results to a sink, but just use the DataSet from the > >> previous step. The disadvantage of this is that you might end up with > >> a too large Flink job, which might also hurt performance. > >> > >> Best, > >> Gábor > >> > >> > >> > >> > >> > >> > >> 2016-09-01 18:09 GMT+02:00 Supun Kamburugamuve <supu...@gmail.com>: > >> > Hi, > >> > > >> > Does Flink support nested iterations? We are trying to develop a > complex > >> > machine learning algorithm which has 3 iterations nested. > >> > > >> > Best, > >> > Supun.. > >> > > >> > > > > > > > > > > > -- > > Supun Kamburugamuve > > Member, Apache Software Foundation; http://www.apache.org > > E-mail: su...@apache.org; Mobile: +1 812 219 2563 > > > > > -- Supun Kamburugamuve Member, Apache Software Foundation; http://www.apache.org E-mail: supun@apache.o <supu...@gmail.com>rg; Mobile: +1 812 219 2563
