Gyula Fora created FLINK-13852:
----------------------------------
Summary: Support storing in-progress/pending files in different
directories (StreamingFileSink)
Key: FLINK-13852
URL: https://issues.apache.org/jira/browse/FLINK-13852
Project: Flink
Issue Type: New Feature
Components: Connectors / FileSystem
Reporter: Gyula Fora
Currently in-progress and pending files are stored in the same directory as the
final output file. This can be problematic depending on the usage of the final
output files. One example would be loading the data to hive where we can only
load all files in a certain directory.
I suggest we allow specifying a Pending/Inprogress base path where we create
the same bucketing structure as the final files to store only the non-final
files.
To support this we need to extend the RecoverableWriter interface with a new
open method for example:
RecoverableFsDataOutputStream open(Path path, Path tmpPath) throws IOException;
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
