Hi Jacek,

Thanks for reporting the issue! Did you hit the same problem when you set
the `spark.jars.ivy` config with Spark 3.5? If this config never worked
with a relative path, we should change the wording in the migration guide.

Thanks,
Wenchen

On Sun, Apr 27, 2025 at 10:27 PM Jacek Laskowski <ja...@japila.pl> wrote:

> Hi,
>
> I found in docs/core-migration-guide.md:
>
> - Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by
> default to isolate the existing systems from Apache Ivy's incompatibility.
> To restore the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`.
>
> With that, I used spark.jars.ivy to run spark-shell with ~/.ivy2 directory
> for local dependencies.
>
> bin/spark-sql --conf spark.jars.ivy=~/.ivy2
>
> This ended up with the following issue:
>
> Exception in thread "main" java.lang.IllegalArgumentException: basedir
> must be absolute: ~/.ivy2/local
> at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48)
> at
> org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137)
> at
> org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44)
> at
> org.apache.spark.util.MavenUtils$.createRepoResolvers(MavenUtils.scala:159)
> at org.apache.spark.util.MavenUtils$.buildIvySettings(MavenUtils.scala:287)
> at
> org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:118)
> at
> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:341)
> at org.apache.spark.deploy.SparkSubmit.org
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:961)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> A workaround is to use an absolute path.
>
> Is this a known issue? Should I report it against rc4? Please guide.
> Thanks!
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on Bluesky <https://bsky.app/profile/books.japila.pl>
>
> <https://twitter.com/jaceklaskowski>
>

Reply via email to