DanilSerd opened a new pull request, #2448: URL: https://github.com/apache/datafusion-ballista/pull/2448
# Which issue does this PR close? Closes #2357. # Rationale for this change The Flight server receives a graceful shutdown notification, but its completion is not tracked by the executor shutdown barrier. The executor can therefore exit while Flight is still streaming results, causing clients to receive connection resets or truncated results. Shutdown-time shuffle cleanup can also delete files before active readers have finished. # What changes are included in this PR? - Track Flight server completion so executor shutdown waits for active Flight streams to drain. Keep the server in `service_handlers` so failures during normal operation still trigger shutdown. - Move final shuffle cleanup after all tracked components finish shutting down. Periodic TTL cleanup and the existing cleanup-enabled condition are unchanged. - Add `final_results_survive_sigterm` to the chaos HA suite. It uses a real scheduler and executor, starts fetching a large SQL result with HTTP/2 backpressure, sends SIGTERM after the first batch, waits for scheduler-observed deregistration, then verifies that all rows arrive and the executor exits successfully. # Are there any user-facing changes? Graceful executor shutdown now waits for active Flight streams instead of terminating them prematurely. When shutdown cleanup is enabled, shuffle files are retained until tracked services finish. This protects requests already in progress; it does not keep the executor available for partitions that clients have not yet requested. No public API or configuration changes. # How was this tested? The regression failed before the fix with a broken connection after receiving 49,152 of 800,000 rows, and passes with the fix. After rebasing onto upstream main: - `cargo build --profile ci --locked -p ballista-chaos --bins` passed. - `cargo test --profile ci --locked -p ballista-executor --lib`: 54 tests passed. - `cargo test --profile ci --locked -p ballista-chaos`: 36 tests passed, including the graceful Flight regression. - `cargo fmt --all -- --check` and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
