Zakelly Lan created FLINK-38574:
-----------------------------------
Summary: Avoid reusing re-uploaded sst files when checkpoint
notification is delayed
Key: FLINK-38574
URL: https://issues.apache.org/jira/browse/FLINK-38574
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.20.3, 2.0.1, 2.1.1
Reporter: Zakelly Lan
Assignee: Zakelly Lan
It might be possible that from the TM's perspective, the checkpoint
notification for last checkpoint is executed after the start of the next
checkpoint, assuming no concurrent checkpoints? If so, this might indicate a
bug. I'm saying that because this may make the
`RocksIncrementalSnapshotStrategy` behaves wrong, diverging from the state
tracking maintained by the `SharedStateRegistry` on the JM side. For example,
considering a buggy event timeline for `RocksIncrementalSnapshotStrategy`:
* Checkpoint 1 finished with state handle A for 1.sst.
* Checkpoint 2 start, based on no checkpoint, so re-upload the 1.sst with
handle B
* Received Checkpoint 1's notification of finish.
* Checkpoint 2 finished. JM thinks the handle A is subsumed.
* Checkpoint 3 start, based on checkpoint 1, for 1.sst we reuse the handle A.
* Received Checkpoint 1's subsume, Checkpoint 2's notification of finish.
* Checkpoint 3 finished. JM received the handle A's placeholder and tell
'Attempt to reference unknown state', since the handle A is removed when
checkpoint 2 finished.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
