Jiwon Park created SPARK-58113:
----------------------------------
Summary: [K8S] Support publishNotReadyAddresses in driver service
Key: SPARK-58113
URL: https://issues.apache.org/jira/browse/SPARK-58113
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 5.0.0
Reporter: Jiwon Park
{{DriverServiceFeatureStep}} creates the driver service as headless
({{clusterIP: None}}). Kubernetes publishes DNS records for a headless service
only for *Ready* pods — the EndpointSlice controller computes endpoint
readiness as {{service.Spec.PublishNotReadyAddresses || (serving &&
!terminating)}}, and the legacy {{tolerate-unready-endpoints}} annotation is
not honored by it.
Users commonly attach a readiness probe to the driver pod through
{{spark.kubernetes.driver.podTemplateFile}} — a typical setup for long-running
Spark Connect servers, where pod readiness gates traffic routing. Probes are
not Spark-managed fields, so they pass through to the driver pod. When such a
probe gates on a port that only binds late in startup — e.g. the Spark Connect
gRPC port, which binds after {{SparkContext}} initialization completes — a
circular dependency arises:
executors need the driver service DNS to register -> DNS requires the driver
pod to be Ready -> Ready requires the probe to pass -> the probe requires
initialization to complete -> initialization keeps replacing executors that die
with {{UnknownHostException}}.
SPARK-32975 already acknowledges this coupling:
{{ExecutorPodsAllocator.start()}} blocks on {{waitUntilReady}} with the comment
_"as the headless service won't be resolvable by DNS until the driver pod is
ready"_. However, that wait runs *inside* {{SparkContext}} initialization,
while a probe on a post-initialization port can only pass *after*
initialization completes — so for this class of deployment the wait can never
succeed, and raising {{spark.kubernetes.allocation.driver.readinessTimeout}}
(the guidance in SPARK-49079) only burns the full timeout before the race
resumes unchanged.
The failure is fatal with static allocation: {{maxNumExecutorFailures = max(3,
2 * spark.executor.instances)}} is 3 for a single executor, and executor churn
during the NotReady window (one failure every ~5s) exceeds it before the 30s
{{maxRegisteredResourcesWaitingTime}} escape. The driver exits with code 11
({{EXCEED_MAX_EXECUTOR_FAILURES}}) on every attempt, so the application can
never start. Observed in production; trivially reproducible with a driver
readiness probe on the Connect port plus {{spark.executor.instances=1}}.
*Proposal*: add {{spark.kubernetes.driver.service.publishNotReadyAddresses}}
(boolean, default {{false}}, preserving current behavior). When enabled,
{{DriverServiceFeatureStep}} sets the corresponding field on the driver service
spec, so DNS records exist from pod creation and executors register over the
already-bound driver RPC port. This also makes the SPARK-32975 wait unnecessary
for such deployments.
The driver service is a single-pod discovery endpoint (executor->driver RPC,
block manager, UI, Spark Connect since SPARK-52193), not a load-balancing
service; publishing not-ready addresses is the standard Kubernetes pattern for
headless discovery services (cf. StatefulSet peer discovery). This follows the
existing precedent of exposing driver service spec fields as configurations
(SPARK-39490: {{ipFamilyPolicy}}/{{ipFamilies}}; {{deleteOnTermination}}).
There is no first-class knob for this today; workarounds require excluding
{{DriverServiceFeatureStep}} via
{{spark.kubernetes.driver.pod.excludedFeatureSteps}} and re-implementing the
service (including the {{spark.driver.host}} wiring) in a custom feature step,
or cluster-level admission webhooks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
