avantgardnerio opened a new pull request, #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088
## Summary
Closes #2071.
Adds a compile-time `BALLISTA_PROTOCOL_VERSION: u32 = 1` in
`ballista_core` that gates executor↔scheduler traffic, plus
Kubernetes-style `/healthz` and `/readyz` probes on both binaries so
live upgrades have something to observe.
**Protocol version**
- New `uint32 ballista_protocol_version = 7` on `ExecutorRegistration`.
- Executor populates it on registration, heartbeat, and `poll_work`.
- Scheduler rejects mismatches (or missing metadata on heartbeat) with
`Status::failed_precondition`, an `info!` log, and increments
`ballista_scheduler_rejected_by_protocol_version_total` (prometheus
feature).
- Executor `/readyz` flips off on RPC failure, and after 5 consecutive
heartbeat failures the executor self-terminates so a mismatched pod
crash-loops until its image is bumped rather than sitting idle.
**Health probes**
- Scheduler: mounted on the existing axum router (same port as gRPC/REST).
`/healthz` is always 200. `/readyz` returns 200 iff at least
`--min-ready-executors` executors are registered (default 1; set to 0
for dev/single-node).
- Executor: new HTTP server on `--bind-health-port` (default `50053`),
since Arrow Flight isn't HTTP. `/healthz` is always 200 — deliberately
independent of scheduler connectivity to prevent restart cascades.
`/readyz` reflects the last heartbeat.
**Rolling upgrade semantics** (documented in `kubernetes.md`)
| scheduler ↔ executor | outcome |
|---|---|
| old ↔ old | works |
| old ↔ new | works (old scheduler ignores the unknown field) |
| new ↔ new | works |
| new ↔ old | rejected — the case the check exists for |
Bump both Deployments' image tags together in the same release; each
rolls independently under its own `maxSurge`, and the scheduler's
`/readyz` gate keeps `Service` traffic on the old scheduler until the
new one has matching-version executors registered.
**Bump policy**: bump `BALLISTA_PROTOCOL_VERSION` only on releases that
change the executor↔scheduler wire format. Most releases won't.
## Test plan
- [x] `cargo test -p ballista-scheduler --lib` (236 pass, incl. 8 new)
- [x] `cargo test -p ballista-executor --lib` (45 pass)
- [x] `cargo clippy --all-targets -p ballista-core -p ballista-executor -p
ballista-scheduler` (clean)
- [x] `cargo fmt --all`
- [x] End-to-end smoke against a 2-executor local cluster: all four
`/healthz` and `/readyz` endpoints return 200; scheduler `/readyz`
shows `ready: 2 executors registered`.
New unit tests cover:
- Register rejected on mismatched protocol version
- Heartbeat rejected on mismatched protocol version
- Heartbeat rejected when metadata is missing entirely
- `poll_work` rejected on mismatched protocol version
- `/healthz` always OK
- `/readyz` fails with no executors
- `/readyz` OK when `min_ready_executors = 0`
- `/readyz` OK after an active heartbeat is recorded
## Notes for reviewers
- The zero default of the new proto field naturally means "old
executor" and is rejected once we start at
`BALLISTA_PROTOCOL_VERSION = 1`. This is intentional per the issue's
"no dual-implementation window" design.
- The old scheduler is permissive because it doesn't know the field
exists — so during a rolling upgrade, old-scheduler ↔ new-executor
keeps working. Only new-scheduler ↔ old-executor is rejected.
- Health probes are unconditional: they mount regardless of the
`rest-api` feature or `--disable-rest-api`, since they're
operational, not part of the REST API surface.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
