Re: High Flink checkpoint Size

2021-06-23 Thread Vijayendra Yadav
kinesis/FlinkKinesisConsumer.java#L158-L159 > > Best, > Yun Tang > From: Vijayendra Yadav > Sent: Wednesday, June 23, 2021 11:02 > To: user > Subject: High Flink checkpoint Size > > Hi Team, > > I have two flink Streaming Jobs > 1) Flink streaming from

Re: High Flink checkpoint Size

2021-06-23 Thread Yun Tang
-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L158-L159 Best, Yun Tang From: Vijayendra Yadav Sent: Wednesday, June 23, 2021 11:02 To: user Subject: High Flink checkpoint Size Hi Team, I

High Flink checkpoint Size

2021-06-22 Thread Vijayendra Yadav
Hi Team, I have two flink Streaming Jobs 1) Flink streaming from KAFKA and writing to s3 2) Fling Streaming from KINESIS (KDS) and writing to s3 Both Jobs have similar checkpoint duration. Job #1 (KAFKA) checkpoint size is only 85KB Job #2 (KINESIS) checkpoint size is 18MB There are no checkpoi