[
https://issues.apache.org/jira/browse/SPARK-59124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59124:
-----------------------------------
Labels: pull-request-available (was: )
> [CORE][K8S] Support spark.kubernetes.files.avoidDownloadSchemes for K8s
> cluster mode
> ------------------------------------------------------------------------------------
>
> Key: SPARK-59124
> URL: https://issues.apache.org/jira/browse/SPARK-59124
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 4.4.0
> Reporter: WonYong-Jang
> Priority: Major
> Labels: pull-request-available
>
> In K8s cluster mode, {{SparkSubmit}} unconditionally downloads
> {{spark.files}} to the driver's
> local disk and rewrites the URIs to
> {{{}[file:/tmp/spark-|file:///tmp/spark-]<uuid>/...{}}}, discarding the
> original
> remote URI (e.g. the {{s3a://}} path under
> {{{}spark.kubernetes.file.upload.path{}}}). The files
> are still delivered to executors via the driver file server, so
> {{SparkFiles.get()}} works
> inside a task, but no single shared path remains that {{spark.read.*}} can
> use.
> SPARK-47475 already made this skippable for jars via
> {{spark.kubernetes.jars.avoidDownloadSchemes}} (4.0.0), and SPARK-55077 is
> doing the same for
> archives. This asks for the same for files, via
> {{{}spark.kubernetes.files.avoidDownloadSchemes{}}}, applying the existing
> {{avoidDownload}}
> predicate to {{args.files}} in {{{}SparkSubmit.prepareSubmitEnvironment{}}}.
> Full description, code references, reproduction and workaround:
> [https://github.com/apache/spark/issues/58310]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
