andygrove commented on PR #2192: URL: https://github.com/apache/datafusion-ballista/pull/2192#issuecomment-5969853566
**[P1] Preserve retired-task reservations after job completion** The retained reservations live inside the execution graph, but retirement now allows the job to finish while redundant tasks are still running. `TaskManager::succeed_job()` then evicts the graph, and subsequent completions contribute **0** in `sum_vcores_for_statuses()`. Under push scheduling, those reserved vcores are never returned; repeated empty-side joins can eventually exhaust scheduler-visible executor capacity. Although post-eviction refund handling is unchanged, this PR makes that path reachable for successfully completed jobs with retired tasks still in flight. I reproduced this by holding a probe task, completing the empty sibling and remaining stages, calling `succeed_job()`, then checking the held completion’s refund: **1 before eviction, 0 afterward**. The current lifecycle test checks the reservation before eviction, so it misses this case. Could we retain outstanding reservation accounting independently of the active graph until completion/cancellation is acknowledged, and extend the regression test to cover completion after successful-job eviction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
