[
https://issues.apache.org/jira/browse/SPARK-58113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiwon Park updated SPARK-58113:
-------------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> [K8S] Support publishNotReadyAddresses in driver service
> --------------------------------------------------------
>
> Key: SPARK-58113
> URL: https://issues.apache.org/jira/browse/SPARK-58113
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 4.3.0
> Reporter: Jiwon Park
> Priority: Major
>
> {{DriverServiceFeatureStep}} creates the driver service as headless
> ({{clusterIP: None}}). Kubernetes publishes DNS records for a headless
> service only for *Ready* pods — the EndpointSlice controller computes
> endpoint readiness as {{service.Spec.PublishNotReadyAddresses || (serving &&
> !terminating)}}, and the legacy {{tolerate-unready-endpoints}} annotation is
> not honored by it.
> Users commonly attach a readiness probe to the driver pod through
> {{spark.kubernetes.driver.podTemplateFile}} — a typical setup for
> long-running Spark Connect servers, where pod readiness gates traffic
> routing. Probes are not Spark-managed fields, so they pass through to the
> driver pod. When such a probe gates on a port that only binds late in startup
> — e.g. the Spark Connect gRPC port, which binds after {{SparkContext}}
> initialization completes — a circular dependency arises:
> executors need the driver service DNS to register -> DNS requires the driver
> pod to be Ready -> Ready requires the probe to pass -> the probe requires
> initialization to complete -> initialization keeps replacing executors that
> die with {{UnknownHostException}}.
> SPARK-32975 already acknowledges this coupling:
> {{ExecutorPodsAllocator.start()}} blocks on {{waitUntilReady}} with the
> comment _"as the headless service won't be resolvable by DNS until the driver
> pod is ready"_. However, that wait runs *inside* {{SparkContext}}
> initialization, while a probe on a post-initialization port can only pass
> *after* initialization completes — so for this class of deployment the wait
> can never succeed, and raising
> {{spark.kubernetes.allocation.driver.readinessTimeout}} (the guidance in
> SPARK-49079) only burns the full timeout before the race resumes unchanged.
> The failure is fatal with static allocation: {{maxNumExecutorFailures =
> max(3, 2 * spark.executor.instances)}} is 3 for a single executor, and
> executor churn during the NotReady window (one failure every ~5s) exceeds it
> before the 30s {{maxRegisteredResourcesWaitingTime}} escape. The driver exits
> with code 11 ({{EXCEED_MAX_EXECUTOR_FAILURES}}) on every attempt, so the
> application can never start. Observed in production; trivially reproducible
> with a driver readiness probe on the Connect port plus
> {{spark.executor.instances=1}}.
> *Proposal*: add {{spark.kubernetes.driver.service.publishNotReadyAddresses}}
> (boolean, default {{false}}, preserving current behavior). When enabled,
> {{DriverServiceFeatureStep}} sets the corresponding field on the driver
> service spec, so DNS records exist from pod creation and executors register
> over the already-bound driver RPC port. This also makes the SPARK-32975 wait
> unnecessary for such deployments.
> The driver service is a single-pod discovery endpoint (executor->driver RPC,
> block manager, UI, Spark Connect since SPARK-52193), not a load-balancing
> service; publishing not-ready addresses is the standard Kubernetes pattern
> for headless discovery services (cf. StatefulSet peer discovery). This
> follows the existing precedent of exposing driver service spec fields as
> configurations (SPARK-39490: {{ipFamilyPolicy}}/{{ipFamilies}};
> {{deleteOnTermination}}).
> There is no first-class knob for this today; workarounds require excluding
> {{DriverServiceFeatureStep}} via
> {{spark.kubernetes.driver.pod.excludedFeatureSteps}} and re-implementing the
> service (including the {{spark.driver.host}} wiring) in a custom feature
> step, or cluster-level admission webhooks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
