Hi Ferenc, Thanks a lot for the proposal. I agree with Martijn. Maybe you can search publicly available sources (e.g., MLs of different projects, stackoverflow, etc), to make sure that there is an organic need for this FS support. That would also help the Flink community to evaluate the proposal easily. WDYT?
Regards, Jeyhun On Fri, Mar 8, 2024 at 11:46 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Ferenc, > > I'm +0: I have seen no demand for Ozone, but if the community is OK with > it, why not. > > Best regards, > > Martijn > > On Mon, Feb 26, 2024 at 6:08 AM Ferenc Csaky <ferenc.cs...@pm.me.invalid> > wrote: > > > Hi, > > > > gentle reminder on this thread, any opinions or thoughts? > > > > Regards, > > Ferenc > > > > > > > > > > On Thursday, February 8th, 2024 at 18:02, Ferenc Csaky > > <ferenc.cs...@pm.me.INVALID> wrote: > > > > > > > > > > > Hello devs, > > > > > > I would like to start a discussion regarding Apache Ozone FS support. > The > > > jira [1] is stale for quite a while, but supporting it with some > > limitations could > > > be done with minimal effort. > > > > > > Ozone do not have truncate() impl, so it falls to the same category as > > > Hadoop < 2.7 [2], on Datastream API it requires the usage of > > > OnCheckpointRollingPolicy when checkpointing enabled to make sure > > > the FileSink will not use truncate(). > > > > > > Table API is a bit trickier, because checkpointing policy cannot be > > ocnfigured > > > explicitly (why?), it behaves differently regarding the write mode [3]. > > Bulk mode > > > is covered, but for fow format, auto-compaction has to be set. > > > > > > Even with the mentioned limitations, I think it would worth to add > > support for OFS, > > > it would require 1 small change to enable "ofs" [4] and documenting the > > limitations. > > > > > > WDYT? > > > > > > Regards, > > > Ferenc > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-28231 > > > [2] > > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general > > > [3] > > > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 > > > [4] > > > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62 > > >
