Re: Issue with incremental checkpointing size

2021-12-07 Thread Caizhi Weng
Hi! Thanks for the clarification. Could you expand checkpoint #195 and #196 for details? Slow end-to-end checkpoint time may be caused by various reasons, for example if the data processing is slow it will backpressure the checkpoint (if that is the case, alignment duration should be high for som

Re: Issue with incremental checkpointing size

2021-12-06 Thread Caizhi Weng
Hi! the checkpointing size is not going beyond 300 MB Is 300MB the total size of checkpoint or the incremental size of checkpoint? If it is the latter one, Flink will only store necessary information (for example the keys and the fields that are selected) in checkpoint and it is compressed, so f

Issue with incremental checkpointing size

2021-12-06 Thread Vidya Sagar Mula
Hi, In my project, we are trying to configure the "Incremental checkpointing" with RocksDB in the backend. We are using Flink 1.11 version and RockDB with AWS : S3 backend Issue: -- In my pipeline, my window size is 5 mins and the incremental checkpointing is happening for every 2 mins. I am