avantgardnerio opened a new pull request, #2289: URL: https://github.com/apache/datafusion-ballista/pull/2289
Two fixes to the Kubernetes deployment guide. Docs only, no code changes. ## 1. Executors dialed the /readyz-gated Service, so the example cluster could not bootstrap The executor Deployment was configured with `--scheduler-host=ballista-scheduler`, the client-facing Service, which respects `/readyz`. The scheduler's `/readyz` returns 503 until at least `--min-ready-executors` (default 1) executors have registered, so on a cold start that Service has no ready Endpoints, executors cannot reach the scheduler to register, and the scheduler never becomes ready. `ballista-scheduler-registration` is already defined in the same manifest for exactly this purpose, with `publishNotReadyAddresses: true`, and the file's own comment describes the chicken-and-egg it exists to prevent. Nothing dialed it. #2088 introduced the `/readyz` gate and readinessProbe; #2245 added the registration Service to solve the resulting bootstrap problem but did not repoint the executor at it. So the remedy already shipped, it just was not connected. ## 2. The scheduler Deployment needs `maxUnavailable: 1` The guide recommended `maxSurge: 25% / maxUnavailable: 0` for both Deployments. That is right for the executors, where it preserves capacity through the roll, and it deadlocks the scheduler. A new scheduler pod is not Ready until an executor registers with it, but running executors hold established connections to the pod being replaced and have no reason to re-register. With `maxUnavailable: 0` Kubernetes cannot retire that outgoing pod until the incoming one is Ready, so the rollout stalls and the cluster keeps serving the old image with nothing reported as failed. The default policy behaves the same way, since 25% of 1 replica rounds down to 0. `maxUnavailable: 1` lets the outgoing pod go first. Its executors lose their connection, re-register through `ballista-scheduler-registration`, and the incoming pod reaches Ready. The cost is a short window with no scheduler, so queries in flight fail. This is not equivalent to `type: Recreate`: with `replicas: 3` Kubernetes still rolls one scheduler at a time and keeps the other two serving, so concurrent schedulers remain possible. Both strategies are now in the example manifest rather than only in prose. ## Validation Fix 2 is confirmed on a 32-executor EKS cluster (Kubernetes 1.34, TPC-H SF1000). Rolling the scheduler to a different image with the default policy stalled on "1 old replicas are pending termination" until timeout while the old pod kept serving. With `maxUnavailable: 1` the same change rolled cleanly twice, reporting `0 of 1 updated replicas are available` and then success, with all 32 executors re-registering against the new pod. Fix 1 is reasoned rather than reproduced: a Service with no ready Endpoints cannot be dialed. It also matches the wiring our working cluster uses, which is how the discrepancy surfaced. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
