Hi Wenchen,
Looks like it didn't work in 3.5 either.
❯ ./bin/spark-shell --version
25/04/28 20:37:48 WARN Utils: Your hostname, Jaceks-Mac-mini.local resolves
to a loopback address: 127.0.0.1; using 192.168.68.100 instead (on
interface en1)
25/04/28 20:37:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.5
/_/
Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 17.0.15
Branch HEAD
Compiled by user ubuntu on 2025-02-23T20:30:46Z
Revision 7c29c664cdc9321205a98a14858aaf8daaa19db2
Url https://github.com/apache/spark
Type --help for more information.
❯ ./bin/spark-shell --conf spark.jars.ivy=~/.ivy2
25/04/28 20:37:54 WARN Utils: Your hostname, Jaceks-Mac-mini.local resolves
to a loopback address: 127.0.0.1; using 192.168.68.100 instead (on
interface en1)
25/04/28 20:37:54 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
Exception in thread "main" java.lang.IllegalArgumentException: basedir must
be absolute: ~/.ivy2/local
at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48)
at
org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137)
at
org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44)
at
org.apache.spark.deploy.SparkSubmitUtils$.createRepoResolvers(SparkSubmit.scala:1274)
at
org.apache.spark.deploy.SparkSubmitUtils$.buildIvySettings(SparkSubmit.scala:1381)
at
org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:182)
at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:339)
at org.apache.spark.deploy.SparkSubmit.org
$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:969)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
We should change the wording in the migration guide and fix it (at least
report it as an issue). I can do both if allowed :wink:
Pozdrawiam,
Jacek Laskowski
----
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on Bluesky <https://bsky.app/profile/books.japila.pl>
<https://twitter.com/jaceklaskowski>
On Mon, Apr 28, 2025 at 5:04 AM Wenchen Fan <cloud0...@gmail.com> wrote:
> Hi Jacek,
>
> Thanks for reporting the issue! Did you hit the same problem when you set
> the `spark.jars.ivy` config with Spark 3.5? If this config never worked
> with a relative path, we should change the wording in the migration guide.
>
> Thanks,
> Wenchen
>
> On Sun, Apr 27, 2025 at 10:27 PM Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> I found in docs/core-migration-guide.md:
>>
>> - Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by
>> default to isolate the existing systems from Apache Ivy's incompatibility.
>> To restore the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`.
>>
>> With that, I used spark.jars.ivy to run spark-shell with ~/.ivy2
>> directory for local dependencies.
>>
>> bin/spark-sql --conf spark.jars.ivy=~/.ivy2
>>
>> This ended up with the following issue:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: basedir
>> must be absolute: ~/.ivy2/local
>> at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48)
>> at
>> org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137)
>> at
>> org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44)
>> at
>> org.apache.spark.util.MavenUtils$.createRepoResolvers(MavenUtils.scala:159)
>> at
>> org.apache.spark.util.MavenUtils$.buildIvySettings(MavenUtils.scala:287)
>> at
>> org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:118)
>> at
>> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:341)
>> at org.apache.spark.deploy.SparkSubmit.org
>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:961)
>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204)
>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227)
>> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96)
>> at
>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> A workaround is to use an absolute path.
>>
>> Is this a known issue? Should I report it against rc4? Please guide.
>> Thanks!
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> "The Internals Of" Online Books <https://books.japila.pl/>
>> Follow me on Bluesky <https://bsky.app/profile/books.japila.pl>
>>
>> <https://twitter.com/jaceklaskowski>
>>
>
