[
https://issues.apache.org/jira/browse/SPARK-58969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kishan Ghetia updated SPARK-58969:
----------------------------------
Component/s: Kubernetes
(was: Bug)
> [K8S] Local dependency uploads lose URI fragment aliases
> --------------------------------------------------------
>
> Key: SPARK-58969
> URL: https://issues.apache.org/jira/browse/SPARK-58969
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.1.2
> Reporter: Kishan Ghetia
> Priority: Major
>
> Spark's Kubernetes submission path drops URI fragments from locally uploaded
> dependencies.
> Spark accepts dependency URIs in the form `source#alias`. The fragment defines
> the application-visible name. For example:
> --files /tmp/query-random.sql#query.sql
> The physical object should be uploaded as `query-random.sql`, while the
> resulting `spark.files` entry should retain `#query.sql`. This allows the
> driver to open `./query.sql` and executors to resolve
> `SparkFiles.get("query.sql")`.
> On Kubernetes, `BasicDriverFeatureStep` passes local FILES, JARS, and
> SUBMIT_PYTHON_FILES through `uploadAndTransformFileUris`, but does not restore
> their fragments afterward. Archive fragments are handled separately and do not
> have this problem.
> This differs from YARN, where the distributed-cache implementation retains the
> URI fragment and uses it as the localized link name.
> The issue can be reproduced by running this test -
> [https://github.com/malware27/spark/commit/0fce035c761a17555b16b16d22141ac3a683e81a]
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
