I think what needs to be done is to implement a org.apache.flink.core.fs.RecoverableWriter for the respective file system. Similar to HadoopRecoverableWriter and S3RecoverableWriter.
Cheers, Till On Thu, May 28, 2020 at 6:00 PM Israel Ekpo <israele...@gmail.com> wrote: > Hi Till, > > Thanks for your feedback and guidance. > > It seems similar work was done for S3 filesystem where relocations were > removed for those file system plugins. > > https://issues.apache.org/jira/browse/FLINK-11956 > > It appears the same needs to be done for Azure File systems. > > I will attempt to connect with Klou today to collaborate to see what the > level of effort is to add this support. > > Thanks. > > > > On Thu, May 28, 2020 at 11:54 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Israel, >> >> thanks for reaching out to the Flink community. As Guowei said, the >> StreamingFileSink can currently only recover from faults if it writes to >> HDFS or S3. Other file systems are currently not supported if you need >> fault tolerance. >> >> Maybe Klou can tell you more about the background and what is needed to >> make it work with other file systems. He is one of the original authors of >> the StreamingFileSink. >> >> Cheers, >> Till >> >> On Thu, May 28, 2020 at 4:39 PM Israel Ekpo <israele...@gmail.com> wrote: >> >>> Guowei, >>> >>> What do we need to do to add support for it? >>> >>> How do I get started on that? >>> >>> >>> >>> On Wed, May 27, 2020 at 8:53 PM Guowei Ma <guowei....@gmail.com> wrote: >>> >>>> Hi, >>>> I think the StreamingFileSink could not support Azure currently. >>>> You could find more detailed info from here[1]. >>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-17444 >>>> Best, >>>> Guowei >>>> >>>> >>>> Israel Ekpo <israele...@gmail.com> 于2020年5月28日周四 上午6:04写道： >>>> >>>>> You can assign the task to me and I will like to collaborate with >>>>> someone to fix it. >>>>> >>>>> On Wed, May 27, 2020 at 5:52 PM Israel Ekpo <israele...@gmail.com> >>>>> wrote: >>>>> >>>>>> Some users are running into issues when using Azure Blob Storage for >>>>>> the StreamFileSink >>>>>> >>>>>> https://issues.apache.org/jira/browse/FLINK-17989 >>>>>> >>>>>> The issue is because certain packages are relocated in the POM file >>>>>> and some classes are dropped in the final shaded jar >>>>>> >>>>>> I have attempted to comment out the relocated and recompile the >>>>>> source but I keep hitting roadblocks of other relocation and filtration >>>>>> each time I update a specific pom file >>>>>> >>>>>> How can this be addressed so that these users can be unblocked? Why >>>>>> are the classes filtered out? What is the workaround? I can work on the >>>>>> patch if I have some guidance. >>>>>> >>>>>> This is an issue in Flink 1.9 and 1.10 and I believe 1.11 has the >>>>>> same issue but I am yet to confirm >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> >>>>>
