Move files read by flink

I am reading files from a folder suppose /files/* Files are pushed into that folder. /files/file1_2018_03_09.csv /files/file2_2018_03_09.csv Flink is reading files from the folder fine but as the no of files grows how do I move the files into another folder? Currently I am using cronjob to mo

Re: Move files read by flink

Is there any way to do that? I have been searching for way to do that but in vain. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Move files read by flink

Where does Flink expose filesystem? Is it from env? or inputstream? TextInputFormat format = new TextInputFormat(new org.apache.flink.core.fs.Path(url.toString())); DataStream inputStream = env.readFile(format,url.toString(),FileProcessingMode.PROCESS_CONTINUOUSLY, 1); DataStream parsedStream =

Re: Move files read by flink

>From path flink thread it seems like there is thread showing this. But just wonder is this resolved?Where to find if final split is complete

Re: Running the executables from Flink distribution of the source build

Can you try to look into log folder for error? May be some clue. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Move files read by flink

so, if you want to have your file parsed try to stay away from flink file parser (v1.4). Use nifi to parse files and then you could use Kafka or Flink. My data pipeline looks like: ftp <-> nifi <-> kafka <-> flink -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/