Re: data sink stops method

2015-10-15 Thread Till Rohrmann
Could you post a minimal example of your code where the problem is reproducible? I assume that there has to be another problem because env.execute should actually trigger the execution. Cheers, Till ​ On Thu, Oct 8, 2015 at 8:58 PM, Florian Heyl wrote: > Hey Stephan and Pieter, > That was the

Re: data sink stops method

2015-10-08 Thread Florian Heyl
Hey Stephan and Pieter, That was the same what I thought, so I simply changed the code like this: original.writeAsCsv(outputPath, "\n", " ", WriteMode.OVERWRITE) env.execute() transformPred.writeAsCsv(outputPath2, "\n", " ", WriteMode.OVERWRITE) env.execute() But he still not execute the two co

Re: data sink stops method

2015-10-08 Thread Stephan Ewen
Yes, sinks in Flink are lazy and do not trigger execution automatically. We made this choice to allow multiple concurrent sinks (spitting the streams and writing to many outputs concurrently). That requires explicit execution triggers (env.execute()). The exceptions are, as mentioned, the "eager"

Re: data sink stops method

2015-10-08 Thread Pieter Hameete
Hi Florian, I believe that when you call *JoinPredictionAndOriginal.collect* the environment will execute your program up until that point. The Csv writes are after this point, so in order to execute these steps I think you would have to call *.execute()* after the Csv writes to trigger the execut

data sink stops method

2015-10-08 Thread Florian Heyl
Hi, I need some help to figure out why one method of mine in a pipeline stops the execution on the hdfs. I am working with the 10.0-SNAPSHOT and the code is the following (see below). The method stops on the hdfs by calling the collect method (JoinPredictionAndOriginal.collect) creating a data s