SEPURI-SAI-KRISHNA opened a new issue, #13208: URL: https://github.com/apache/seatunnel/issues/13208
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. #10997 has the same symptom, but no task failed or restored there. No open PR changes `CheckpointBarrierTriggerOperation`. #12314 changes how the coordinator handles a failed barrier RPC; it does not cover this case, because here the RPC succeeds. ### What happened On Zeta, a batch job whose pipeline restores after a task failure can stay `RUNNING` forever. Nothing fails, and nothing is written after the restore. The cause is a checkpoint barrier from the failed attempt that is delivered to the restored attempt. `CheckpointBarrierTriggerOperation` sends a barrier from the checkpoint coordinator to the source enumerator. On the worker it looks up the target task group. When it is not there, it retries every 2 s, up to 30 times, while `taskIsEnded` is false: ```java exception -> exception instanceof TaskGroupContextNotFoundException && !server.taskIsEnded(taskLocation.getTaskGroupLocation()) ``` A restore resets the task group state to `CREATED`, which is not an end state, and deploys a new task group at the same `TaskGroupLocation`. So a barrier that misses the failed task group keeps retrying, and reaches the new one: 1. The source finishes, and the coordinator triggers the final barrier (`Long.MAX_VALUE`, `COMPLETED_POINT_TYPE`). 2. The sink fails at the same moment. The barrier finds no task group, and the pipeline state is reset to `CREATED`, so the operation retries. 3. 3 s later the restored task group is deployed. The next retry finds it and triggers the old final barrier on it. 4. The restored tasks take it as the final barrier, and the reader stops before it reads anything. The restored coordinator has no such pending checkpoint, so it drops every acknowledgement (`skip already ack checkpoint id: 9223372036854775807`). The final checkpoint never completes, and the job never ends. Whether the job hangs depends on timing. If the stale barrier arrives after the restored reader has already read its rows, the job goes on, and only the barrier is wrong. ### How I checked it The first run was the Docker E2E on dev plus a GraphQL connector change; all others were on dev `b7ef4936fe`, in process with `AbstractSeaTunnelServerTest`, JDK 17. - **Found in a real job.** While changing `GraphQLIT`, a FakeSource (two tables) to GraphQL sink job on Zeta failed every row, because the server rejected the mutation. After the first restore the job stayed `RUNNING` for over an hour. The coordinator triggered the final barrier at 15:17:58.871, while the sink task group was failing. Exactly 4 s later (two retries), at 15:18:02.887, three acknowledgements for checkpoint `9223372036854775807` arrived from the restored tasks, and were skipped. The restored reader never produced a row. - **Again in CI, with no test hook.** In the fork run https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42597013855 (job `updated-modules-integration-test-part-1 (11, ubuntu-latest)`, the same GraphQL change), the GraphQL source job `graphql_rejected_query_to_assert.conf` on Zeta failed and was restored 3 times within 10 s. 3 s after the third restore, at 10:26:14.260, three acknowledgements for checkpoint `9223372036854775807` were skipped. The job then stayed `RUNNING` for 5 hours, until the CI job was cancelled at its 300 minute limit. - **Stock connectors.** FakeSource (two tables, one row each) to the `InMemory` test sink with `throw_runtime_exception_list`: 1 of 4 jobs hung with the same log lines, and the other 3 failed after 3 restores. The race is narrow; in later runs 0 of 56 hung. - **The operation on its own, deterministic.** I set the task group state to `CREATED`, sent the operation for a task group that is not deployed, and deployed a task group at that location 1 s later. The new task group received the barrier, 3 of 3. A barrier for a running task group is delivered as normal. - **A whole job with the window widened.** A test sink fails 500 ms after its first write, and the final barrier is held for 1.5 s on the worker before it is handled, as if the RPC were slow. In 18 of 18 attempts the first try found no task group in state `CREATED`, and every one of them was then delivered to a newer deployment. 1 of 9 jobs hung. ### Proposed fix Do not retry a missing task group in `CheckpointBarrierTriggerOperation`. A deployment publishes its task group context before it starts its tasks, and every task (enumerator, reader and sink, committer) then waits in `READY_START` until the coordinator has seen `READY_START` from every task of the pipeline and calls `notifyTaskStart`. So by the time any barrier exists, every task group of that deployment is registered on its worker. A missing task group means that deployment has ended, and its barrier can be dropped. The coordinator does not act on these RPC results today, and with #12314 it ignores a failure whose pending checkpoint is no longer current. On the worker, Hazelcast logs the dropped operation at ERROR with its stack trace, as it already does on `dev` when the task group has ended; I can turn that into one INFO line if you prefer. With that change in the same setups: - the deterministic case drops the stale barrier, with `TaskGroupContextNotFoundException`, and still delivers to a running task group; - 6 of 6 widened-window jobs end `FAILED` after 3 restores, with no stale acknowledgements; - normal batch jobs still finish, and the failing jobs still fail after 3 restores; - 83 tests in 13 engine-server test classes pass, and one more is disabled in the source: `CheckpointCoordinatorTest`, `SavePointTest`, `SavePointBusySourceTest`, `CheckpointErrorRestoreEndTest`, `CheckpointTimeOutTest`, `CheckpointBarrierTriggerErrorTest`, `TaskExecutionServiceTest`, `JobMasterMasterFailoverResourceTest` and others. A stricter alternative is to carry the target execution id in the operation and drop the barrier on a mismatch. That also covers an operation delayed past the restore, but the master would have to keep the execution id of each deployment, which it does not today, including after a master failover. I would start with the smaller change unless you prefer this one. Other task operations use the same retry (added in #3009). Most of them cannot cross a restore, because the sending task waits for them with `join()`: its task group cannot end while the operation is pending, and the restore waits for every task group to end. I checked this with `AssignSplitOperation`: holding it on the worker held up the restore until it returned. `BarrierFlowOperation` is sent the same way. The barrier trigger is different, because the coordinator does not wait for it. Two groups could cross in theory, and I have not seen either: the reader-to-enumerator operations (`SourceNoMoreElementOperation`, `SourceRegisterOperation`, `RequestSplitOperation`, `RestoredSplitOperation`), which are sent with an interruptible `get()`, and the other coordinator-to-task operations (`NotifyTaskStartOperation`, `NotifyTaskRestoreOperation`, `CheckpointFinishedOperation`, `CheckpointEndOperation`), which no task waits for. A stale `NotifyTaskStartOperation` would let a restored tas k start before its pipeline is ready, which the reasoning above relies on not happening. The execution id alternative would cover all of them. I would look at them separately. I would like to submit a PR with the fix and a unit test. ### SeaTunnel Version dev (b7ef4936fe) ### SeaTunnel Config ```conf env { parallelism = 1 job.mode = "BATCH" } source { FakeSource { tables_configs = [ { schema = { table = "t1", fields { id = int, name = string } } rows = [ { kind = INSERT, fields = [1, "a"] } ] }, { schema = { table = "t2", fields { id = int, name = string } } rows = [ { kind = INSERT, fields = [2, "b"] } ] } ] } } sink { # Any sink whose write fails while the source finishes. In the test I used the # InMemory test sink with throw_runtime_exception_list. InMemory { throw_runtime_exception_list = ["e0", "e1", "e2", "e3", "e4"] } } ``` ### Running Command ```shell # InMemory is a test connector, so I ran the job in process through # AbstractSeaTunnelServerTest#startJob and waited for the job result. ``` ### Error Exception ```log CheckpointCoordinator - received restore CheckpointCoordinator with alreadyStarted: false SubPlan - Wait 3s and then restore the pipeline Job ... FakeSourceSplitEnumerator - Assigning splits to readers 0 [...] CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807 CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807 CheckpointCoordinator - skip already ack checkpoint id: 9223372036854775807 (nothing more; the job stays RUNNING) ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version JDK 17 (in process). The GraphQL run used the JDK 11 engine image. ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
