Thanks Guowei. Good to know it may have been fixed in another way. I'll have a try. BTW, which version is this kind of sink supported from? Does it also work for batch mode?
On Mon, Apr 25, 2022 at 2:42 PM Guowei Ma <guowei....@gmail.com> wrote: > Hi > > Afaik the commit files action happens at the committer operator instead of > the JM size after the new sink api [1]. > > It means this would not happen if you use the new `FlinkSink`.[2] > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > > [2] > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java > > Best, > Guowei > > > On Sun, Apr 24, 2022 at 11:39 AM Yi Tang <ssnailt...@gmail.com> wrote: > >> >> >> ---------- Forwarded message --------- >> From: Yi Tang <ssnailt...@gmail.com> >> Date: Sun, Apr 24, 2022 at 11:29 AM >> Subject: Write to Aliyun OSS via FileSystem connector hang Job Master on >> Finishing >> To: <user@flink.apache.org> >> >> >> Hi team; >> >> I'm trying to write to aliyun oss via FileSystem connector. The job >> master always hangs on finishing. >> >> Looks like it is because the FileSystem connector commits the files by >> #finalizeGlobal while the Job is finishing, which includes some rename >> operations. However, the aliyun oss FileSystem renames files by copying, >> which seems expensive. >> >> Any suggestions about this scenario? >> >> Thanks and regards. >> >>
