andygrove opened a new issue, #2686:
URL: https://github.com/apache/datafusion-ballista/issues/2686
### Background
#2192 moves push-scheduled vcore reservations into a ledger in `TaskManager`
(`outstanding_task_reservations`), so a reservation is refunded exactly once,
even after the job graph has been evicted.
### Problem
In `SchedulerState::launch_tasks` (`ballista/scheduler/src/state/mod.rs`),
when `executor_manager.get_executor_metadata(&executor_id)` fails, the batch's
reservations are taken from the ledger but nothing is refunded:
```rust
Err(e) => {
error!("Failed to launch new task, could not get executor metadata:
{e}");
state
.task_manager
.take_task_reservations(&executor_id, &reservations);
(vec![], HashSet::new())
}
```
Unlike the launch-error branch above it, this branch doesn't remove the
executor. If the metadata lookup fails while the executor is still registered
(for example, a transient error from a non-memory `ClusterState` backend), the
vcores bound for that batch stay reserved until the executor re-registers.
Before #2192, this branch refunded the batch's task count.
### Suggested fix
Refund what `take_task_reservations` returns, the same way the
rejected-launch branch does:
```rust
let freed = state
.task_manager
.take_task_reservations(&executor_id, &reservations);
(vec![(executor_id.clone(), freed)], HashSet::new())
```
Add a test next to the launch-rejection tests in `state/mod.rs` that binds
tasks to an executor whose metadata lookup fails and checks that the returned
slots equal the bound vcores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
