Re: Checkpoint expired before completing with cleanupInRocksdbCompactFilter

2019-05-09 Thread Congxian Qiu
Hi, Mu Is there anything  looks like `Received  late message for now expired checkpoint attempt ${checkpointID} from ${taskkExecutionID} of job ${jobID}` in JM log? If yes, that means this task complete the checkpoint too long (maybe receive barrier too late, maybe spend too much time to do che

Checkpoint expired before completing with cleanupInRocksdbCompactFilter

2019-05-08 Thread Mu Kong
Hi community, I'm glad that in Flink 1.8.0, it introduced cleanupInRocksdbCompactFilter to support state clean up for rocksdb backend. We have an application that heavily relies on managed keyed store. As we are using rocksdb as the state backend, we were suffering the issue of ever-growing state