Re: Flink CheckPoint/Savepoint Behavior Question

2021-02-02 Thread Arvid Heise
Hi Jason, you got it perfectly right. So everything that is not in an explicit state (or checkpointed in CheckpointedFunction#snapshotState) is lost on recovery. However, Flink applications always go through the complete life-cycle. Note that I'd look into CheckpointedFunction if the side-informa

Re: Flink CheckPoint/Savepoint Behavior Question

2021-02-01 Thread Raghavendar T S
Flink is aware of all the tasks running in the cluster. If any of the tasks fails, the failed task is restored using the checkpoint (only If the task uses Flink Operator State). This scenario will not use savepoints. Savepoints are same as checkpoints and the difference is that the savepoints are c

Flink CheckPoint/Savepoint Behavior Question

2021-02-01 Thread Jason Liu
We currently have some logic to load data from S3 into memory in our Flink/Kinesis Analytics app. This happens before the RichFunction.open() function. We have two questions here and I can't find too much information in the apache.org website: 1. (More of a clarification) When Flink does c