andygrove opened a new issue, #2691:
URL: https://github.com/apache/datafusion-ballista/issues/2691
### Background
#2686 / #2689 refund the vcore reservations when
`SchedulerState::launch_tasks` can't get executor metadata. The vcores come
back, but the tasks in that batch don't.
### Problem
At bind time, `ExecutionGraph::pop_next_task` (and the `bind_task_*` paths)
already record each task as `Running { executor_id }` in the stage's
`task_infos`. On the metadata-miss branch of `launch_tasks`, those tasks are
never launched. They aren't reported as failed, and their stage isn't reset.
Nothing will ever send a status update for them, so the job hangs unless that
executor is later lost and its stage gets reset.
Since the refund sets `if_revive`, a lookup that keeps failing also makes
each revive bind more pending partitions to the same executor. Those get
stranded the same way, until nothing is left to bind.
### Suggested fix
On a metadata miss, return the batch's tasks to pending, the way an
executor-lost reset does for tasks on that executor. Or, if that executor is
genuinely gone, treat it as lost so the existing reset path runs. Add a test
that binds tasks, drops the metadata, launches, and then checks that the
stage's tasks are pending again and get rescheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
