Hi, as the documentation for the StreamingFileSink indicates [1], it is required to enable checkpoints if you want to use bulk encoded output formats atm.
I'm not sure whether there are concrete plans to change this behaviour in the future because it breaks with exactly once processing guarantees. Klou might know more. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html Cheers, Till On Mon, Jul 22, 2019 at 10:21 PM Elkhan Dadashov <elkhan.dadas...@gmail.com> wrote: > Hi Flink Dev team, > > Will StreamingFileSink.forBulkFormat(...) support overriding > OnCheckpointRollingPolicy? > > Does anyone use StreamingFileSink *with checkpoint disabled *for writing > Parquet output files? > > The output parquet files are generated, but they are empty, and stay in > *inprogress* state, even when the job completes: > > .part-0-0.inprogress.3e31ba42-588c-48cc-ad6d-d0ebcf1d8632 > .part-1-0.inprogress.78e1f1dc-3c1c-417b-8270-2bf0298f985a > .part-2-0.inprogress.087cf3f1-7e2d-4a03-a518-62f576ed7eea > > Exactly-once semantics is not important for my case, would then using > *BucketingSink* is the only option ? > > Thanks. >
