Hi Fred, I think this is the current behavior (though it would be helpful to know which version of Flink you’re using).
From an email conversation with Kostas in January of this year: > Hi Ken, Jingsong and Li, > > Sorry for the late reply. > > As Jingsong pointed out, upon calling close() the StreamingFileSink > does not commit the in-progress/pending files. > The reason for this is that the close() method of any UDF including > sink functions is called on both normal termination and termination > due to failure. > Given this, we cannot commit the files, because in case of failure > they should be reverted. > > Actually we are currently updating the StreamingFileSink docs to > includes this among other things. > Also the differentiation between normal termination and termination > due to failure will hopefully be part of Flink 1.11 and > this is the FLIP to check > https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs > > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs>. > > Cheers, > Kostas Though it looks like FLIP-46 is still under discussion, and thus 1.11 doesn’t have a fix for this? — Ken > On Sep 7, 2020, at 8:39 AM, Teunissen, F.G.J. (Fred) <fred.teunis...@ing.com > <mailto:fred.teunis...@ing.com>> wrote: > > Hi All, > > My flink-job is using bounded input sources and writes the results to a > StreamingFileSink. > When it has processed all the input the job is finished and closes. But the > output files are still > named “<prefix>-0-0.<ext>.inprogress.<guid>”. I expected them to be named > ““<prefix>-0-0.<ext>”. > > Did I forget some setting or something else? > > Regards, > Fred > > ----------------------------------------------------------------- > ATTENTION: > The information in this e-mail is confidential and only meant for the > intended recipient. If you are not the intended recipient, don't use or > disclose it in any way. Please let the sender know and delete the message > immediately. > ----------------------------------------------------------------- -------------------------- Ken Krugler http://www.scaleunlimited.com <http://www.scaleunlimited.com/> custom big data solutions & training Hadoop, Cascading, Cassandra & Solr -------------------------- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr
