Hi Morgan, Regarding your first question, if the Kafka connector is configured to use exactly-once semantics it will check the offsets of partitions when recovering from the checkpoint, so there will be no data loss or duplication. I'm not quite sure I understood the second part of the first question regarding the HPA, are you asking if Flink is actively triggering a new checkpoint before the pod shutdown? If so then the answer is no, it will use the latest completed checkpoint.
For the second question, it will use the latest checkpoint, there is no 'active'/forced creation of a new checkpoint before the re-scaling. You can find some more information about the mechanism here: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/ Best regards, Alexander On Wed, Dec 15, 2021 at 10:24 PM Geldenhuys, Morgan Karl < morgan.geldenh...@tu-berlin.de> wrote: > Greetings, > > > I would like to find out more about Flink's new reactive mode as well as > the rescaling feature regarding fault tolerance. For the following question > lets assume checkpointing is enabled using HDFS. > > > So first question, if I have a job where the source(s) and sink(s) are > configured to use Kafka with exactly-once processing enabled, how does > reactive mode handle this? On reconfigurations using the Horizontal Pod > Autoscaler, does it recover to the latest checkpoint? > > > Second question, for re-scaling, does it automatically create a > savepoint and then rescale or does it use the latest checkpoint to ensure > results are consistent during reconfiguration. > > > Thank you in advance! > > > M. > -- Alexander Preuß | Junior Engineer - Data Intensive Systems alexanderpre...@ververica.com <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, Jinwei (Kevin) Zhang
