Maybe the User list does not have knowledge about this. That's why I also resend on the Dev list. Sorry for cross posting
Hi All, I have a stream aggregation job which reads from Kafka and writes some Sinks. When I submit my job Flink checkpoint size keeps increasing if I use unaligned checkpoint settings and it does not emit any window results. If I use an aligned checkpoint, size is somewhat under control(still big) but Checkpoint alignment takes a long time. I would like to implement something similar [1]. I believe if UnboundedSourceWrapper pause reading future watermark partitions it will reduce the size of the checkpoint and I can use unaligned checkpointing. What do you think about this approach ? Do you have another solution ? One more question: I was reading code to implement the above idea. I saw this code [2] Does Flink Runner have a similar implementation? Thanks [1] https://github.com/apache/flink/pull/11968 [2] https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java#L207
