Hi, I found in docs/core-migration-guide.md:
- Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by default to isolate the existing systems from Apache Ivy's incompatibility. To restore the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`. With that, I used spark.jars.ivy to run spark-shell with ~/.ivy2 directory for local dependencies. bin/spark-sql --conf spark.jars.ivy=~/.ivy2 This ended up with the following issue: Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ~/.ivy2/local at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48) at org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137) at org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44) at org.apache.spark.util.MavenUtils$.createRepoResolvers(MavenUtils.scala:159) at org.apache.spark.util.MavenUtils$.buildIvySettings(MavenUtils.scala:287) at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:118) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:341) at org.apache.spark.deploy.SparkSubmit.org $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:961) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) A workaround is to use an absolute path. Is this a known issue? Should I report it against rc4? Please guide. Thanks! Pozdrawiam, Jacek Laskowski ---- "The Internals Of" Online Books <https://books.japila.pl/> Follow me on Bluesky <https://bsky.app/profile/books.japila.pl> <https://twitter.com/jaceklaskowski>
