Hi Ferenc, I'm +0: I have seen no demand for Ozone, but if the community is OK with it, why not.
Best regards, Martijn On Mon, Feb 26, 2024 at 6:08 AM Ferenc Csaky <ferenc.cs...@pm.me.invalid> wrote: > Hi, > > gentle reminder on this thread, any opinions or thoughts? > > Regards, > Ferenc > > > > > On Thursday, February 8th, 2024 at 18:02, Ferenc Csaky > <ferenc.cs...@pm.me.INVALID> wrote: > > > > > > > Hello devs, > > > > I would like to start a discussion regarding Apache Ozone FS support. The > > jira [1] is stale for quite a while, but supporting it with some > limitations could > > be done with minimal effort. > > > > Ozone do not have truncate() impl, so it falls to the same category as > > Hadoop < 2.7 [2], on Datastream API it requires the usage of > > OnCheckpointRollingPolicy when checkpointing enabled to make sure > > the FileSink will not use truncate(). > > > > Table API is a bit trickier, because checkpointing policy cannot be > ocnfigured > > explicitly (why?), it behaves differently regarding the write mode [3]. > Bulk mode > > is covered, but for fow format, auto-compaction has to be set. > > > > Even with the mentioned limitations, I think it would worth to add > support for OFS, > > it would require 1 small change to enable "ofs" [4] and documenting the > limitations. > > > > WDYT? > > > > Regards, > > Ferenc > > > > [1] https://issues.apache.org/jira/browse/FLINK-28231 > > [2] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general > > [3] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 > > [4] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62 >
