SEPURI-SAI-KRISHNA opened a new pull request, #13209: URL: https://github.com/apache/seatunnel/pull/13209
Yes, glad to contribute to Apache SeaTunnel. ### Purpose of this pull request Closes #13208. `CheckpointBarrierTriggerOperation` retried for up to 60 s when the target task group was missing and its state was not an end state. A pipeline restore resets the state to `CREATED` and deploys a new task group at the same `TaskGroupLocation`, so a barrier that missed the failed task group was triggered on the restored one. With the final barrier (`Long.MAX_VALUE`), the restored reader stopped before reading, the restored coordinator skipped every acknowledgement, and a batch job stayed `RUNNING` forever. The issue has the logs and the measurements. The operation now looks the task group up once and fails with `TaskGroupContextNotFoundException` when it is not there. That is safe because no task starts before every task of its pipeline has reported `READY_START` and the coordinator has called `notifyTaskStart`, and a deployment publishes its context before it starts its tasks. So when a barrier exists, its target task group is registered, and a missing one has ended. The coordinator does not act on the RPC result (and #12314, if merged, ignores a failure whose pending checkpoint is no longer current). Only this operation changes. `BarrierFlowOperation` and the other task operations keep their retry: their senders wait with `join()`, so the sending task group cannot end, and the pipeline cannot restore, while they are pending. The issue lists the operations that could cross a restore in theory. ### Does this PR introduce _any_ user-facing change? Yes. A batch job that failed and restored at the moment its source finished could stay `RUNNING` forever; it now restores normally and ends `FINISHED` or `FAILED`. No option, default, API or state format changes. On the worker, a dropped stale barrier is logged at ERROR by Hazelcast, as on `dev` when the task group has ended. ### How was this patch tested? - `TaskExecutionServiceTest.testBarrierForAMissingTaskGroupDoesNotReachTheNextDeployment`: the state is `CREATED`, the barrier is sent for a task group that is not deployed, and a task group is deployed at that location 1 s later. The barrier must not reach it, and the operation must fail with `TaskGroupContextNotFoundException`. On `dev` the barrier reaches the new task group (`expected: <0> but was: <1>`). - `testBarrierReachesARunningTaskGroup`: a barrier for a running task group is still delivered. - `TaskExecutionServiceTest` 22 of 22 with the fix, 21 of 22 without it (only the new stale test fails). 13 engine-server test classes, 83 tests, pass, including `CheckpointCoordinatorTest`, `SavePointTest`, `SavePointBusySourceTest`, `CheckpointErrorRestoreEndTest`, `CheckpointTimeOutTest`, `CheckpointBarrierTriggerErrorTest` and `JobMasterMasterFailoverResourceTest`. `spotless:check` passes. - Whole jobs in process: with the final barrier held 1.5 s on the worker and a sink that fails 500 ms after its first write, `dev` delivered the barrier to a newer deployment in 18 of 18 attempts and 1 of 9 jobs hung; with this change 6 of 6 jobs ended `FAILED` after 3 restores with no stale acknowledgements. Normal batch jobs (in-memory, 5 splits with parallelism 2, two tables) still finish, and failing batch and streaming jobs still fail after 3 restores. - No E2E is added: the race needs the barrier to miss its task group within a few milliseconds of a failure, which an E2E cannot force without a timing hook. The unit test reproduces the mechanism deterministically. Fork run: https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42588706253 passes, apart from `connector-file-sftp-it`. That job fails because the `atmoz/sftp:alpine-3.7` image is gone from Docker Hub (#13202, fixed by #13203); it is unrelated to this change and fails on `dev` too. ### Check list * [X] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) (no new jar) * [X] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs (not needed: no option or behavior a user configures) * [X] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR (not needed: no incompatible change) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
