Hi team! We have observed strange behavior when using KafkaSource and have checkpointing enabled.
Even if we do not get checkpoint failures or any means of errors, we see the committed offset going up and down in a crazy manner instead of moving up to reach the end offset of that X topic. We know that Flink does not rely on Kafka for committed offsets, etc, but it captures that important info in the checkpoints it makes. Flink updates Kafka as a means of reporting/monitoring tool only. So in our example, we hava kafkaSource, and we consume from a topic A where it has endOffset: 60_000. The committedOffset after some time reaches the 30_000, and then 40_000, etc. At some point, it reaches the 47_000 and then drops back down not in a fashioned pattern, eg, 47_000 -> 23_000 -> 27_000 -> 36_000 -> 22_000 and so on and so forth. Unfortunately, there is not any error or a checkpoint failure so that we could explain that behavior. Moreover, we would like to try unaligned checkpoint to opt-out if backpressure is doing that harm, but that is out of options at the moment. What do you suggest? Have you experienced that before? Thanks in advance.
