Re: Controlling the amount of checkpoint files

2019-06-15 Thread Boris Lublinsky
So if you have externalized checkpoints, they are never purged? The issue is that if your state size is rather large, this seems to be the only option. Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Jun 15, 2019, at 3:39 AM, Robert Metzger wrote: >

Re: Controlling the amount of checkpoint files

2019-06-15 Thread Robert Metzger
Hey Boris, I think the problem is that you are using externalized checkpoints: env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION) Your checkpoints are retained in both failure and cancellation cases, so the checkpoint files with grow indef

Re: Controlling the amount of checkpoint files

2019-06-11 Thread Congxian Qiu
Hi Boris For the configure you gave, you can try to reduce the parallelism of the operator which contains states. Best, Congxian Boris Lublinsky 于2019年6月10日周一 下午9:43写道: > Here is code enabling checkpointing > > // Enable checkpointing > env.enableCheckpointing(6 ) // 1 min > env.getCheck