Thanks Gabor. I was thinking about starting separate jobs. Is there any plans to support nested loops in the future?
Thanks, Supun.. On Thu, Sep 1, 2016 at 12:28 PM, Gábor Gévay <[email protected]> wrote: > Hello Supun, > > Unfortunately, nesting of Flink's iteration constructs are not > supported at the moment. > > There are some workarounds though: > > 1. You can start a Flink job for each step of the iteration. Starting > a Flink job has some overhead, so this only works if there is a > sufficient amount of work in each iteration step. Moreover, this has > the disadvantage that the intermediate results are always need to be > written out and then read back between steps, which might have a > considerable performance impact. > > 2. If you have just a small fixed number of steps, then you can have a > for loop that "unrolls" all the iteration steps, and creates one large > Flink job. The code will be somewhat similar to the first approach, > but you don't call execute between the steps, and you don't write > intermediate results to a sink, but just use the DataSet from the > previous step. The disadvantage of this is that you might end up with > a too large Flink job, which might also hurt performance. > > Best, > Gábor > > > > > > > 2016-09-01 18:09 GMT+02:00 Supun Kamburugamuve <[email protected]>: > > Hi, > > > > Does Flink support nested iterations? We are trying to develop a complex > > machine learning algorithm which has 3 iterations nested. > > > > Best, > > Supun.. > > > > > -- Supun Kamburugamuve Member, Apache Software Foundation; http://www.apache.org E-mail: [email protected] <[email protected]>rg; Mobile: +1 812 219 2563
