phillipleblanc opened a new issue, #2349: URL: https://github.com/apache/datafusion-ballista/issues/2349
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Ballista exposes shared cluster state, job state, and job acquisition interfaces, but these interfaces are not sufficient for a library consumer to recover running jobs after a scheduler fails. Completing recovery currently requires access to private scheduler state and execution graph internals. This issue tracks the public APIs required to implement scheduler high availability outside Ballista. **Describe the solution you would like** Add public APIs for the following capabilities: - [ ] **Execution graph persistence:** Provide versioned encoding and decoding for `ExecutionGraphBox`. - [ ] **Recovered-job activation:** Allow a restored execution graph to be added to the active job set and made available for scheduling. - [ ] **Ownership fencing:** Return an ownership token when a job is acquired and require it when saving job state, assigning tasks, and accepting task status. Updates from an earlier owner must be rejected. - [ ] **Executor failover:** Allow executors to reconnect to another scheduler and retry task status updates that were not acknowledged. - [ ] **Owner-aware control operations:** Allow status, cancellation, and cleanup requests to reach the scheduler that owns the job, or return enough information for the consumer to route them. Completion criteria: - An external crate can implement active-passive scheduler failover using only public Ballista APIs. - A running query can continue after its scheduler process stops. - Recovery is tested with adaptive query execution enabled and disabled. - Updates from the previous scheduler owner are rejected. - Task status updates are not lost during executor reconnection. - Job status and cancellation work through the surviving scheduler. - Incompatible execution graph versions produce a clear error. - A public example demonstrates the complete recovery flow. The following are outside the scope of this issue: - A built-in consensus or membership system - Selection of a durable state backend - Shared shuffle or durable result storage - Automatic high availability in the default scheduler binary **Describe alternatives you have considered** Consumers can maintain a Ballista fork or depend on private scheduler internals. Both approaches create version-specific integrations that cannot be supported as library APIs. **Additional context** This work is required before #2347 can document a complete scheduler recovery implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
