Hi Vijay, To be honest, an 18MB checkpoint size in total might not be something serious. If you really want to dig what inside, you could use Checkpoints#loadCheckpointMetadata [1] to load the _metadata to see anything unexpected.
And you could refer to FlinkKafkaConsumerBase#unionOffsetStates [2] and FlinkKinesisConsumer#sequenceNumsToRestore to compare different operator state stored in kafka and kinesis connector. [1] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java#L99 [2] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L201 [3] https://github.com/apache/flink/blob/10146366bec7feca85acedb23184b99517059bc6/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L158-L159 Best, Yun Tang ________________________________ From: Vijayendra Yadav <contact....@gmail.com> Sent: Wednesday, June 23, 2021 11:02 To: user <user@flink.apache.org> Subject: High Flink checkpoint Size Hi Team, I have two flink Streaming Jobs 1) Flink streaming from KAFKA and writing to s3 2) Fling Streaming from KINESIS (KDS) and writing to s3 Both Jobs have similar checkpoint duration. Job #1 (KAFKA) checkpoint size is only 85KB Job #2 (KINESIS) checkpoint size is 18MB There are no checkpoint failures. But I want to understand why Kinesis streaming has such a huge checkpoint size, is there a way to handle it differently? and reduce the size. Thanks, Vijay
