Re: Table SQL Filesystem CSV recursive directory traversal

2020-11-13 Thread Danny Chan
In the current master code base, all the FileInputFormat default add the files recursively with the given paths. (e.g. the #addFilesInDir method). So it should be supported as default for SQL. Timo Walther 于2020年11月9日周一 下午11:25写道: > Hi Ruben, > > by looking at the code, it seems you should be a

Re: Table SQL Filesystem CSV recursive directory traversal

2020-11-09 Thread Timo Walther
Hi Ruben, by looking at the code, it seems you should be able to do that. At least for batch workloads we are using org.apache.flink.formats.csv.CsvFileSystemFormatFactory.CsvInputFormat which is a FileInputFormat that supports the mentioned configuration option. The problem is that this mig

Table SQL Filesystem CSV recursive directory traversal

2020-11-09 Thread Ruben Laguna
Is it possible? For Dataset I've found [1] : parameters.setBoolean("recursive.file.enumeration", true); // pass the configuration to the data sourceDataSet logs = env.readTextFile("file:///path/with.nested/files") .withParameters(parameters); But can I achieve somethin