Jiangjie Qin created FLINK-16986:
------------------------------------
Summary: Enhance the OperatorEvent handling guarantee during
checkpointing.
Key: FLINK-16986
URL: https://issues.apache.org/jira/browse/FLINK-16986
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Reporter: Jiangjie Qin
When the {{CheckpointCoordinator}} takes a checkpoint, the checkpointing order
is following:
# {{CheckpointCoordinator}} triggers checkpoint on each {{OperatorCoordinator}}
# Each {{OperatorCoordinator}} takes a snapshot.
# Right after taking the snapshot, the {{CheckpointCoordinator}} sends a
{{CHECKPOINT_FIN}} marker through the {{OperatorContext}}.
# Once the {{OperatorContext}} sees {{CHECKPOINT_FIN}} marker, it will wait
for all the previous events are acked and suspend the event gateway to the
operators by buffering the future {{OperatorEvents}} sent from the
{{OperatorCoordinator}} to the operators without actually sending them out.
# The {{CheckpointCoordinator}} waits until all the {{OperatorCoordinator}}s
finish step 2-4 and then triggers the task snapshots.
# The suspension of an event gateway to an operator can be lifted after all
the subtasks of that operator has finished their task checkpoint.
The mechanism above guarantees all the {{OperatorEvents}} sent before taking
the operator coordinator snapshot are handled by the operator before the task
snapshots are taken.
An operator can use this mechanism to know whether an {{OperatorEvent}} it sent
to the coordinator is included in the upcoming checkpoint or not. What it has
to do is to ask the operator coordinator to ACK that OperatorEvent. If the ACK
is received before the operator takes the next snapshot, that OperatorEvent
must have been handled and checkpointed by the OperatorCoordinator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
