Hi, When I submit my spark job, I see the following runtime exception in the
log,
Exception in thread "Thread-1" java.lang.NoClassDefFoundError:
org/apache/spark/streaming/kafka/KafkaUtils
at SparkHdfs.run(SparkHdfs.java:56)
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.streaming.kafka.KafkaUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)I am including
the assembly and streaming jars during submit,
./spark-submit
--jars/Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar,/Users/hadoop/spark-1.2.1/external/kafka/target/spark-streaming-kafka_2.10-1.2.1.jar--class
KafkaMain --master spark://krishs-mbp:7077 --deploy-mode
cluster--num-executors 100 --driver-memory 1g --executor-memory 1g
--supervisefile:///Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
