February 2015 in the Flink community

2015-03-02 Thread Kostas Tzoumas
Hi everyone February might be the shortest month of the year, but the community has been pretty busy: - Flink 0.8.1, a bugfix release has been made available - The project added a new committer - Flink contributors developed a Flink adapter for Apache SAMOA - Flink committers contributed to Go

Re: Getting the name of a file in a directory

2015-03-02 Thread Stephan Ewen
Hi Pietro! The path of an input format is the directory from which all its input is read. It does not contain the specific paths of the contained files. The specific path is part of the "FileInputSplit", which describes a subtask of work (like a file, or a part of a file). If you want the path of

Getting the name of a file in a directory

2015-03-02 Thread pietro
I am reading files from a directory with this statement: /val text = env.readFile(new MyInputFormat(), "/path/to/input/dir/")/ /MyInputFormat/ extends /DelimitedInputFormat/ which extends /DelimitedInputFormat/. In the output Record, I need to add a field that stores the name (or full path) of t

Re: Optimizing degree of parallelism

2015-03-02 Thread Fabian Hueske
Just to clarify. The pull request 410 does not optimize the degree of parallelism of operators with respect to performance / gain. It just sets the maximum possible parallelization for the current execution environment. Cheers, Fabian 2015-03-02 17:20 GMT+01:00 Max Michels : > Hi! > > There is a

Re: Optimizing degree of parallelism

2015-03-02 Thread Max Michels
Hi! There is a pending pull request for this feature. If that is what you had in mind: https://github.com/apache/flink/pull/410 Best regards, Max On Mon, Mar 2, 2015 at 5:11 PM, Alexander Alexandrov wrote: > AFAIK at the moment this is not supported but at the TU Berlin we have a > master stud

Re: Optimizing degree of parallelism

2015-03-02 Thread Alexander Alexandrov
AFAIK at the moment this is not supported but at the TU Berlin we have a master student working on this feature, so it might be possible within the next 3-6 months. Regards, Alexander 2015-03-02 17:01 GMT+01:00 Malte Schwarzer : > Hi everyone, > > I read that Flink is supposed to automatically o

Optimizing degree of parallelism

2015-03-02 Thread Malte Schwarzer
Hi everyone, I read that Flink is supposed to automatically optimize the degree of parallelism. But I never saw any change of parallelism in the web interface without defining dop manually (-p parameter). Is there any of this optimization actually happening? Or how can I switch it on? Cheers Ma