Hi Stefan,

thanks for your answer.

> I think there is a field in FileInputFormat (which TextInputFormat is 
> subclassing) that could serve your purpose if you override the default:

That was my first guess as well. I use the basic setup from 
org.apache.flink.api.java.io.TextInputFormatTest.java and call 
setNestedFileEnumeration(true), but once the stream is processed only the 
content of the .csv file in the top-most folder is printed. The example is just 
a few lines of self-contained code, see below. Does anybody have an idea?

Cheers,
Lukas


import org.apache.flink.api.java.io.TextInputFormat;
import org.apache.flink.core.fs.Path;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;

public class ReadDirectorySSCCE {
    public static void main(String[] args) throws Exception {
        // create given dirs and add a .csv file to each one
        String[] dirs = new String[] {"tmp", "tmp/first/", "tmp/second/"};
        for (String dir: dirs) {
            // create input file
            File tmpDir = new File(dir);
            if (!tmpDir.exists()) {
                tmpDir.mkdirs();
            }
            File tempFile = File.createTempFile("file", ".csv", tmpDir);
            BufferedWriter w = new BufferedWriter(new FileWriter(tempFile));
            w.write("content of " + dir + "/file.csv");
            w.close();
            tempFile.deleteOnExit();
        }
        File root = new File("tmp");

        TextInputFormat inputFormat = new TextInputFormat(new 
Path(root.toURI().toString()));
        inputFormat.setNestedFileEnumeration(true);

        StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
        env.createInput(inputFormat).print();
        env.execute();
    }


> On 7 Dec 2016, at 17:44, Stefan Richter <[email protected]> wrote:
> 
> Hi,
> 
> I think there is a field in FileInputFormat (which TextInputFormat is 
> subclassing) that could serve your purpose if you override the default:
> 
> /**
>  * The flag to specify whether recursive traversal of the input directory
>  * structure is enabled.
>  */
> protected boolean enumerateNestedFiles = false;
> As for compression, I think this class also provides a 
> InflaterInputStreamFactory to read compressed data.
> 
> Best,
> Stefan
> 
>> Am 07.12.2016 um 12:10 schrieb Lukas Kircher <[email protected] 
>> <mailto:[email protected]>>:
>> 
>> Hi all,
>> 
>> I am trying to read nested .csv files from a directory and want to switch 
>> from a custom SourceFunction I implemented to the TextInputFormat. I have 
>> two questions:
>> 
>> 1) Somehow only the file in the root directory is processed, nested files 
>> are skipped. What am I missing? See the attachment for an SSCCE. I get the 
>> same result with flink 1.1.3 no matter if I run it via the IDE or submit the 
>> job to the standalone binary. The file input splits are all there, yet they 
>> don't seem to be processed.
>> 
>> 2) What is the easiest way to read compressed .csv files (.zip)?
>> 
>> Thanks for your help, cheers
>> Lukas
>> 
>> <ReadDirectorySSCCE.java>
> 

Reply via email to