Hi, gentle reminder on this thread, any opinions or thoughts?
Regards, Ferenc On Thursday, February 8th, 2024 at 18:02, Ferenc Csaky <ferenc.cs...@pm.me.INVALID> wrote: > > > Hello devs, > > I would like to start a discussion regarding Apache Ozone FS support. The > jira [1] is stale for quite a while, but supporting it with some limitations > could > be done with minimal effort. > > Ozone do not have truncate() impl, so it falls to the same category as > Hadoop < 2.7 [2], on Datastream API it requires the usage of > OnCheckpointRollingPolicy when checkpointing enabled to make sure > the FileSink will not use truncate(). > > Table API is a bit trickier, because checkpointing policy cannot be ocnfigured > explicitly (why?), it behaves differently regarding the write mode [3]. Bulk > mode > is covered, but for fow format, auto-compaction has to be set. > > Even with the mentioned limitations, I think it would worth to add support > for OFS, > it would require 1 small change to enable "ofs" [4] and documenting the > limitations. > > WDYT? > > Regards, > Ferenc > > [1] https://issues.apache.org/jira/browse/FLINK-28231 > [2] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general > [3] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 > [4] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62
