Hi, Greg
"--add-exports java.base/sun.nio.ch=ALL-UNNAMED " does not need to be added
when SPARK-33772 is completed, so in order to answer your question, I need more
details for testing:
1. Where can I download Java 17 (Temurin-17+35)?
2. What test commands do you use?
Yang Jie
在 2022/6/23 12:54，“Greg Kopff”<g...@q10stats.com> 写入:
Hi.
According to the release notes[1], and specifically the ticket Build and
Run Spark on Java 17 (SPARK-33772)[2], Spark now supports running on Java 17.
However, using Java 17 (Temurin-17+35) with Maven (3.8.6) and
maven-surefire-plugin (3.0.0-M7), when running a unit test that uses Spark
(3.3.0), it fails with:
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module @0x1e7ba8d9) cannot access class sun.nio.ch.DirectBuffer (in
module java.base) because module java.base does not export sun.nio.ch to
unnamed module @0x1e7ba8d9
The full stack is:
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module @0x1e7ba8d9) cannot access class sun.nio.ch.DirectBuffer (in
module java.base) because module java.base does not export sun.nio.ch to
unnamed module @0x1e7ba8d9
at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at
org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
at
org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
[…]
There is a recent StackOverflow question "Java 17 solution for Spark -
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.storage.StorageUtils"[3], which was asked only 2 months ago,
but this pre-dated the Spark 3.3.0 release, and thus predated official support
for Java 17. The solution proposed there results in us adding this
configuration to the Surefire plugin:
<configuration>
<argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>
</configuration>
And, yes, this works.
Now, I understand what this flag achieves … without it the JVM module
system won’t allow Spark to use the sun.nio.ch.DirectBuffer class. My question
is if the requirement to add this flag is currently documented somewhere? I
couldn’t find its and it’s likely to start affecting people when they switch to
Java 17. Right now the web is mostly full of suggestions to use an earlier
version of Java.
Cheers,
—
Greg.
[1]: https://spark.apache.org/releases/spark-release-3-3-0.html
[2]: https://issues.apache.org/jira/browse/SPARK-33772
[3]: https://stackoverflow.com/questions/72230174
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
