[
https://issues.apache.org/jira/browse/SPARK-58649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-58649:
-------------------------------------
Assignee: Yannick Goetschel
> Allow SparkApplication drivers ingress to SparkCluster workers
> ----------------------------------------------------------------
>
> Key: SPARK-58649
> URL: https://issues.apache.org/jira/browse/SPARK-58649
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.0.0
> Environment:
> Reporter: Yannick Goetschel
> Assignee: Yannick Goetschel
> Priority: Major
> Labels: pull-request-available
>
> We run per-user Spark Connect sessions as SparkApplications attached to a
> shared
> standalone SparkCluster. Most jobs are fine, but every so often one dies
> like this:
> {code:java}
> Job aborted due to stage failure: Task 11 in stage 2.0 failed 4 times, most
> recent
> failure: Lost task 11.3 in stage 2.0 (TID 86) (<executor-ip> executor 3):
> TaskResultLost (result lost from block manager)
> {code}
> We run per-user Spark Connect sessions as SparkApplications attached to a
> shared
> standalone SparkCluster. Most jobs are fine, but every so often one dies like
> this:
> {code:java}
> Job aborted due to stage failure: Task 11 in stage 2.0 failed 4 times, most
> recent
> failure: Lost task 11.3 in stage 2.0 (TID 86) (<executor-ip> executor 3):
> TaskResultLost (result lost from block manager)
> {code}
> The driver log explains why only some of them:
> {code:java}
> java.io.IOException: Connecting to /<executor-ip>:<ephemeral-port> timed out
> (120000 ms)
> at
> org.apache.spark.network.client.TransportClientFactory.createClient(...)
> at
> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(...)
> at
> org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(...)
> at org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(...)
> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(...)
> {code}
> The driver cannot reach the executors' block manager. That only matters once
> a task
> result goes over {{spark.task.maxDirectResultSize}} (1 MB): below it the
> result rides
> along inline in the RPC, above it the executor parks the result in its block
> manager and
> hands the driver an {{IndirectTaskResult}} to go and fetch. The tasks that
> stayed under
> the limit never noticed anything. The three that went over timed out, retried
> three
> times and gave up, and the stage aborted on the fourth attempt.
> The culprit is the worker NetworkPolicy the operator generates for every
> SparkCluster in
> {{SparkClusterResourceSpec#buildWorkerNetworkPolicy}}:
> {code:yaml}
> podSelector:
> matchLabels: {spark-role: worker, spark.operator/spark-cluster-name:
> <cluster>}
> ingress:
> - from:
> - podSelector:
> matchLabels: {spark.operator/spark-cluster-name: <cluster>}
> policyTypes: [Ingress]
> {code}
> Only pods carrying the cluster label may talk to the workers. But a
> SparkApplication
> attached to the cluster is a separate resource: its driver is labelled
> {{spark-role: driver}} and {{spark.operator/spark-app-name: <app>}}, never
> the cluster
> label. So the driver is locked out of its own executors.
> The executor-to-driver direction works perfectly well —
> executors register, tasks get dispatched, tasks run. Only the return path for
> large
> results is broken, so the cluster looks healthy right up until a job happens
> to cross
> the 1 MB threshold.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
