Mohamed Amine ABDESSEMED created FLINK-2672:
-----------------------------------------------
Summary: Add partitioned output format to HDFS RollingSink
Key: FLINK-2672
URL: https://issues.apache.org/jira/browse/FLINK-2672
Project: Flink
Issue Type: Improvement
Components: Streaming Connectors
Affects Versions: 0.10
Reporter: Mohamed Amine ABDESSEMED
Priority: Minor
An interesting use case of the HDFS Sink is to dispatch data into multiple
directories depending of attributes present in the source data.
For example, for some data with a timestamp and a status fields, we want to
write it into different directories using a pattern like :
/somepath/%{timestamp}/%{status}
The expected results are somethings like:
/somepath/some_timestamp/wellformed
/somepath/some_timestamp/malformed
/somepath/some_timestamp/incomplete
...
etc
To support this functionality the bucketing and checkpointing logics need to be
changed.
Note: For now, this can be done using the current version of the Rolling HDFS
Sink with the help of splitting data streams and having multiple HDFS sinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
