Hi there,
My name is Philip, a master’s student at Imperial College London. I’m trying to
use Spark to complete my course work assignment. I ran the following code:
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
and got the following error message:
Py4JJavaError: An error occurred while calling
None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class
org.sparkproject.jetty.http.MimeTypes
at
org.sparkproject.jetty.server.handler.gzip.GzipHandler.<init>(GzipHandler.java:190)
at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:485)
at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:147)
at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:147)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:147)
at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:486)
at
org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:486)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:486)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:829)
While in my Macbook’s terminal, it’s showing following error messages:
WARNING: An illegal reflective access operation has occurred
pyspark_mongodb_nb | WARNING: Illegal reflective access by
org.apache.spark.unsafe.Platform
(file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to
constructor java.nio.DirectByteBuffer(long,int)
pyspark_mongodb_nb | WARNING: Please consider reporting this to the
maintainers of org.apache.spark.unsafe.Platform
pyspark_mongodb_nb | WARNING: Use --illegal-access=warn to enable warnings of
further illegal reflective access operations
pyspark_mongodb_nb | WARNING: All illegal access operations will be denied in
a future release
pyspark_mongodb_nb | 21/06/24 06:57:17 WARN NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
pyspark_mongodb_nb | Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
pyspark_mongodb_nb | Setting default log level to "WARN".
pyspark_mongodb_nb | To adjust logging level use sc.setLogLevel(newLevel). For
SparkR, use setLogLevel(newLevel).
pyspark_mongodb_nb | 21/06/24 06:57:20 WARN MacAddressUtil: Failed to find a
usable hardware address from the network interfaces; using random bytes:
bd:af:a7:b4:a2:46:2a:28
I’m wondering if you could help me resolve the issues I have with my laptop. I
have a 2020 MacBook Pro with a M1 chip. Thank you so much in advance.
Best,
Philip Hsu
