SEPURI-SAI-KRISHNA commented on code in PR #13209:
URL: https://github.com/apache/seatunnel/pull/13209#discussion_r4528794613
##########
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/operation/CheckpointBarrierTriggerOperation.java:
##########
@@ -67,41 +64,37 @@ protected void readInternal(ObjectDataInput in) throws
IOException {
barrier = in.readObject();
}
+ /**
+ * Hands the barrier to the target task on this worker.
+ *
+ * <p>No task of a pipeline starts before every task has reported that it
is ready, so when a
+ * barrier exists, its target task group is already registered on this
worker. If it cannot be
+ * found, that deployment has ended. The barrier is not retried: a
pipeline restore deploys a
+ * new task group at the same location, and a retried barrier would reach
that deployment, which
+ * belongs to a new checkpoint coordinator and never completes this
checkpoint.
+ */
@Override
public void runInternal() throws Exception {
SeaTunnelServer server = getService();
- RetryUtils.retryWithException(
- () -> {
- Task task =
- server.getTaskExecutionService()
-
.getExecutionContext(taskLocation.getTaskGroupLocation())
- .getTaskGroup()
- .getTask(taskLocation.getTaskID());
- task.getExecutionContext()
- .getTaskExecutionService()
- .asyncExecuteFunction(
- taskLocation.getTaskGroupLocation(),
- () -> {
- try {
- log.debug(
-
"CheckpointBarrierTriggerOperation [{}]",
- taskLocation);
- task.triggerBarrier(barrier);
- } catch (Exception e) {
- task.getExecutionContext()
- .sendToMaster(
- new
CheckpointErrorReportOperation(
-
taskLocation, e));
- }
- });
- return null;
- },
- new RetryUtils.RetryMaterial(
- Constant.OPERATION_RETRY_TIME,
- true,
- exception ->
- exception instanceof
TaskGroupContextNotFoundException
- &&
!server.taskIsEnded(taskLocation.getTaskGroupLocation()),
- Constant.OPERATION_RETRY_SLEEP));
+ Task task =
+ server.getTaskExecutionService()
Review Comment:
Thanks @nzw921rx. I agree that identifying the deployment is the right
direction, and the complete fix. What I found in the code:
**What the retry covers.** The loop runs on the worker after the operation
has arrived, and only retries `TaskGroupContextNotFoundException` while the
task group state is not ended. Network errors show up on the master's
invocation future, not inside this loop. For a barrier of the current attempt,
the context is never "not deployed yet":
- the worker publishes the context before it starts any task
(`TaskExecutionService.java:731-738`), and a task reports `READY_START` from
that run and waits until the master has recorded it
(`AbstractTask.reportTaskStatus`);
- periodic checkpoints and savepoints need `isAllTaskReady`, which is set
only after every subtask reported `READY_START`, and a restore clears the old
statuses (`cleanPendingCheckpoint`);
- final and schema change checkpoints are requested by a running task of
that attempt;
- after a master failover, checkpoints restart at once only when every
coordinator vertex is `RUNNING`
(`SubPlan.isCheckpointCoordinatorAlreadyStarted`).
So a missing context means the barrier is stale (its deployment has ended),
and a retry can only deliver it to the next deployment at the same location
(#13208). The fork CI (unit tests, all engine ITs) passed without it.
**Identifying the restored task group.** Each deploy already gets a new
`executionId` (`PhysicalVertex.getTaskGroupImmutableInformation`), and the
worker keeps it in `TaskGroupContext`. What is missing:
1. the master keeps the id of each task group's current deployment, also in
an IMap (like `ownedSlotProfilesIMap`), so a new master knows it after failover;
2. `CheckpointBarrierTriggerOperation` carries the expected id (its
serialized form changes);
3. the worker delivers the barrier only when `context.getExecutionId()`
matches and drops it otherwise, so retrying while the context is missing
becomes safe.
This also closes the case DanielLeens raised (a barrier delayed past the
redeploy), and the same check fits the other checkpoint operations sent to
tasks with this retry (`NotifyTaskStartOperation`,
`NotifyTaskRestoreOperation`, `CheckpointFinishedOperation`,
`CheckpointEndOperation`).
Either way works for me:
- (a) merge this small fix, then I open an issue and a PR for the
`executionId` check on all these operations;
- (b) I add the check for the barrier to this PR and keep the retry.
Which do you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]