Thanks Jing for briging this to dicuss. I agree it's not a blocker for graduting the FileSink to @PublicEvolving since the Sink which is the rootcause has marked as @PublicEvolving. But I do also share the same concern with Galen. At least it should be a blocker for removing StreamingFileSink. Btw, seems it's really a big headache for migrating to Sink, we may need to pay more attention to this ticket and try to fix it.
Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Galen Warren" <ga...@cvillewarrens.com.INVALID> 收件人: "dev" <dev@flink.apache.org> 发送时间: 星期五, 2023年 6 月 23日 下午 7:47:24 主题: Re: [DISCUSS] Graduate the FileSink to @PublicEvolving Thanks Jing. I can only offer my perspective on this, others may view it differently. If FileSink is subject to data loss in the "stop-on-savepoint then restart" scenario, that makes it unusable for me, and presumably for anyone who uses it in a long-running streaming application and who cannot tolerate data loss. I still use the (deprecated!) StreamingFileSink for this reason. The bigger picture here is that StreamingFileSink is deprecated and will presumably ultimately be removed, to be replaced with FileSink. Graduating the status of FileSink seems to be a step along that path; I'm concerned about continuing down that path with such a critical issue present. Ultimately, my concern is that FileSink will graduate fully and that StreamingFileSink will be removed and that there will be no remaining option to reliably stop/start streaming jobs that write to files without incurring the risk of data loss. I'm sure I'd feel better about things if there were an ongoing effort to address this FileSink issue and/or a commitment that StreamingFileSink would not be removed until this issue is addressed. My two cents -- thanks. On Fri, Jun 23, 2023 at 1:47 AM Jing Ge <j...@ververica.com.invalid> wrote: > Hi Galen, > > Thanks for the hint which is helpful for us to have a clear big picture. > Afaiac, this will not be a blocking issue for the graduation. There will > always be some (potential) bugs in the implementation. The API is very > stable from 2020. The timing is good to graduate. WDYT? > Furthermore, I'd like to have more opinions. All opinions together will > help the community build a mature API graduation process. > > Best regards, > Jing > > On Tue, Jun 20, 2023 at 12:48 PM Galen Warren > <ga...@cvillewarrens.com.invalid> wrote: > > > Is this issue still unresolved? > > > > https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-30238 > > > > Based on prior discussion, I believe this could lead to data loss with > > FileSink. > > > > > > > > On Tue, Jun 20, 2023, 5:41 AM Jing Ge <j...@ververica.com.invalid> > wrote: > > > > > Hi all, > > > > > > The FileSink has been marked as @Experimental[1] since Oct. 2020. > > > According to FLIP-197[2], I would like to propose to graduate it > > > to @PublicEvloving in the upcoming 1.18 release. > > > > > > On the other hand, as a related topic, FileSource was marked > > > as @PublicEvolving[3] 3 years ago. It deserves a graduation discussion > > too. > > > To keep this discussion lean and efficient, let's focus on FlieSink in > > this > > > thread. There will be another discussion thread for the FileSource. > > > > > > I was wondering if anyone might have any concerns. Looking forward to > > > hearing from you. > > > > > > > > > Best regards, > > > Jing > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java#L129 > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > [3] > > > > > > > > > https://github.com/apache/flink/blob/4006de973525c5284e9bc8fa6196ab7624189261/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/FileSource.java#L95 > > > > > >
