> The BucketingSink suffers from the same problem. It's caused by the fact > that we don't do a "final" checkpoint before shutting down a pipeline. > We're trying to resolve that with FLIP-147 [1].
I thought this was waiting on FLIP-46 -- Graceful Shutdown Handling -- and in fact, the StreamingFileSink is mentioned in that FLIP as a motivating use case. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs On Tue, Oct 13, 2020 at 1:01 PM Aljoscha Krettek <aljos...@apache.org> wrote: > On 13.10.20 11:18, David Anderson wrote: > > I think the pertinent question is whether there are interesting cases > where > > the BucketingSink is still a better choice. One case I'm not sure about > is > > the situation described in docs for the StreamingFileSink under Important > > Note 2 [1]: > > > > ... upon normal termination of a job, the last in-progress files > will > > not be transitioned to the “finished” state. > > > > I know this confuses and frustrates users, but I don't know if the > > BucketingSink has any advantages in this regard. > > The BucketingSink suffers from the same problem. It's caused by the fact > that we don't do a "final" checkpoint before shutting down a pipeline. > We're trying to resolve that with FLIP-147 [1]. > > [1] https://cwiki.apache.org/confluence/x/mw-ZCQ > >
