akshaychitneni opened a new pull request, #2677:
URL: https://github.com/apache/datafusion-ballista/pull/2677
# Which issue does this PR close?
Closes #.
# Rationale for this change
When a running stage is rolled back (a fetch failure, or losing an executor
whose output it reads), it re-runs every task. But each finished task had
already pushed its output locations to the next stage's inputs (or, for a
final
stage, to the job's output) when it reported, and those were kept. After
the
re-run those partitions are recorded twice, so the query returns duplicated
rows with no error.
It happens whenever a task of the stage succeeds before another fails: rare
when every task reads from the lost executor, but routine when each task
reads a
single upstream partition.
# What changes are included in this PR?
- `rollback_running_stage` now drops the stage's entries from its linked
stages'
inputs, or from the job's output locations for a final stage, before the
stage
re-runs.
- Two execution-graph tests: a final stage and an intermediate stage, each
rolled
back after a partial success. Both fail without the fix.
- Chaos Scenario K: kill a map-side executor after some final-aggregate
tasks
have finished; the result must match the baseline.
The adaptive graph keeps stage outputs in its planner and does not re-run a
rolled-back stage yet, so it is unchanged.
# Are there any user-facing changes?
No, other than queries that hit this case no longer return duplicated rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
