DanilSerd opened a new issue, #2357: URL: https://github.com/apache/datafusion-ballista/issues/2357
**Describe the bug** When an executor receives SIGTERM or Ctrl-C while a client is fetching data through Arrow Flight, the executor exits before the active stream finishes. The client receives a stream or transport error and cannot finish reading the result. This is only observable when no tasks are currently running. If a task is active, #2356 causes shutdown to hang before reaching Flight server shutdown, masking this issue. **To Reproduce** 1. Start a scheduler and executor. 2. Run a query that produces a large result. 3. Wait for execution tasks to finish and begin consuming the result slowly through Flight. 4. Send SIGTERM or Ctrl-C to the executor while the Flight stream is active. 5. Observe that the stream terminates before all batches are received. **Expected behavior** The executor should stop accepting new Flight requests, allow active Flight streams to complete, and then exit. **Additional context** The Flight server uses tonic's `serve_with_shutdown`, which supports draining existing connections. However, its `JoinHandle` is not awaited after shutdown begins and is not included in `ShutdownNotifier` completion tracking. As a result, `start_executor_process` can return while Flight is still streaming. The executor's Tokio runtime is then dropped, cancelling the active stream. Relevant code: - `ballista/executor/src/executor_process.rs` - `flight_server_task` - `ShutdownNotifier` - `ballista/executor/src/flight_service.rs` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
