장원용 created SPARK-59124:
---------------------------
Summary: Support spark.kubernetes.files.avoidDownloadSchemes for
K8s cluster mode
Key: SPARK-59124
URL: https://issues.apache.org/jira/browse/SPARK-59124
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Core
Affects Versions: 5.0.0
Reporter: 장원용
Reported at: https://github.com/apache/spark/issues/58310
h2. Summary
In K8s cluster mode, \{{SparkSubmit}} unconditionally downloads
\{{spark.files}} to the driver's
local disk and rewrites the URIs to \{{file:/tmp/spark-<uuid>/...}}, discarding
the original
remote URI (e.g. the \{{s3a://}} path under
\{{spark.kubernetes.file.upload.path}}). The files
are still delivered to executors via the driver file server, so
\{{SparkFiles.get()}} works
inside a task, but no single shared path remains that \{{spark.read.*}} can use.
SPARK-47475 already made this skippable for jars via
{\{spark.kubernetes.jars.avoidDownloadSchemes}} (4.0.0), and SPARK-55077 is
doing the same for
archives. This asks for the same for files, via
{\{spark.kubernetes.files.avoidDownloadSchemes}}, applying the existing
\{{avoidDownload}}
predicate to \{{args.files}} in \{{SparkSubmit.prepareSubmitEnvironment}}.
Full description, code references, reproduction and workaround:
https://github.com/apache/spark/issues/58310
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
