Hello devs, I would like to start a discussion regarding Apache Ozone FS support. The jira [1] is stale for quite a while, but supporting it with some limitations could be done with minimal effort.
Ozone do not have truncate() impl, so it falls to the same category as Hadoop < 2.7 [2], on Datastream API it requires the usage of OnCheckpointRollingPolicy when checkpointing enabled to make sure the FileSink will not use truncate(). Table API is a bit trickier, because checkpointing policy cannot be ocnfigured explicitly (why?), it behaves differently regarding the write mode [3]. Bulk mode is covered, but for fow format, auto-compaction has to be set. Even with the mentioned limitations, I think it would worth to add support for OFS, it would require 1 small change to enable "ofs" [4] and documenting the limitations. WDYT? Regards, Ferenc [1] https://issues.apache.org/jira/browse/FLINK-28231 [2] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general [3] https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 [4] https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62
