Re: Parallel read text

2016-05-30 Thread Robert Metzger
Hi David, I guess you can verify it by adding custom log statements into the Flink code (therefore, you need to recompile Flink). Maybe a debugger is also sufficient (if you are running Flink locally). We are currently reworking the reading of static files for the streaming environment. Maybe its

Re: Parallel read text

2016-05-28 Thread David Olsen
Thank you for the advice! Now I have a new question. I read the source[1] streaming env exploits FileSourceFunction, which inherits RichParallelSourceFunction, to create split input[2]. I know I can set parallelism in streaming env, but any way I can verify that at runtime the split files or the f

Re: Parallel read text

2016-05-28 Thread Chesnay Schepler
ExecutionEnvironment.readTextFile will read the file in parallel. On 28.05.2016 09:59, David Olsen wrote: After searching on the internet I still do not find the answer (with key word like 'apache flink parallel read text') I am looking for. So asking here before jumping to write code ... My