[
https://issues.apache.org/jira/browse/SPARK-59553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59553:
-----------------------------------
Labels: pull-request-available (was: )
> Make SparkSession.addArtifact honor spark.jars.ivySettings
> ----------------------------------------------------------
>
> Key: SPARK-59553
> URL: https://issues.apache.org/jira/browse/SPARK-59553
> Project: Spark
> Issue Type: Task
> Components: Connect, SQL
> Affects Versions: 4.2.0
> Environment: SparkSession.addArtifact supports resolving Maven
> coordinates through ivy:// URIs, but it does not currently honor
> `spark.jars.ivySettings`.
> For classic Spark, Artifact.newIvyArtifacts always builds the default Ivy
> settings. For Spark Connect, Ivy coordinates are resolved by the client and
> the resulting JARs are uploaded to the server. Consequently, the client's Ivy
> configuration and network access are used instead of the server's
> spark.jars.ivySettings.
> This differs from other Maven-resolution surfaces such as spark-submit and
> SparkContext.addJar. It also prevents Spark Connect from using repositories,
> mirrors, credentials, or repository policies configured only on the server.
> Proposed changes:
> * Make SparkSession.addArtifact and addArtifacts use spark.jars.ivySettings
> when resolving ivy:// coordinates.
> * Move Spark Connect Ivy resolution from the client to the server:
> * Add a protocol representation for an unresolved Ivy coordinate.
> * Have the client send the coordinate instead of resolving and uploading
> its JARs.
> * Resolve the coordinate on the server and register the resulting JARs
> through the existing artifact-management path.
> * Preserve URI ordering and deduplicate shared transitive dependencies.
> * Apply the behavior to both addArtifact and addArtifacts.
> Compatibility and fallback:
> * When spark.jars.ivySettings is configured, load it and use it for
> resolution.
> * When it is not configured, retain the existing default Ivy resolver
> behavior.
> * A client should use server-side resolution only when the server
> advertises support. With an older server, retain the existing client-side
> resolution and upload behavior.
> * Older clients must continue to work with newer servers.
> * Preserve the existing handling of per-coordinate ?repos= repositories. If
> these cannot safely be resolved by the server, fall back to the existing
> client-side path.
> * A failure to load or use an explicitly configured spark.jars.ivySettings
> should be reported rather than silently bypassing it and resolving from
> default repositories.
> Reporter: Qianlan Chen
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
