Completely diffrent than the one I set: Classpath is ::/tmp/spark/spark-branch-1.1/conf:/tmp/spark/spark-branch-1.1/assembly/target/scala-2.10/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-core-3.2.2.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-rdbms-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib_managed/jars/datanucleus-api-jdo-3.2.1.jar
On Tue, Dec 16, 2014 at 7:18 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > > Can you open the file bin/spark-class and then put an echo $CLASSPATH > below the place where they exports it and see what are the contents? > On 16 Dec 2014 22:46, "Daniel Haviv" <danielru...@gmail.com> wrote: > >> I've added every jar in the lib dir to my classpath and still no luck: >> >> >> CLASSPATH=/tmp/spark/spark-branch-1.1/lib/datanucleus-api-jdo-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib/datanucleus-core-3.2.2.jar:/tmp/spark/spark-branch-1.1/lib/datanucleus-rdbms-3.2.1.jar:/tmp/spark/spark-branch-1.1/lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib/spark-bagel_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-bagel_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-catalyst_2.10-1.1.2-SNAPSHOT-tests.jar:/tmp/spark/spark-branch-1.1/lib/spark-core_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-core_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples-1.1.2-SNAPSHOT-hadoop2.3.0.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-examples_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-graphx_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-graphx_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive-thriftserver_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-hive-thriftserver_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-mllib_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-mllib_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-repl_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-repl_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-sql_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-sql_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming_2.10-1.1.2-SNAPSHOT-tests.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume-sink_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-flume-sink_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-kafka_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-kafka_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-mqtt_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-mqtt_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-twitter_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-twitter_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-zeromq_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-streaming-zeromq_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-tools_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-tools_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/spark-yarn_2.10-1.1.2-SNAPSHOT.jar:/tmp/spark/spark-branch-1.1/lib/spark-yarn_2.10-1.1.2-SNAPSHOT-sources.jar:/tmp/spark/spark-branch-1.1/lib/TestSerDe.jar >> >> >> >> >> On Tue, Dec 16, 2014 at 7:05 PM, Akhil Das <ak...@sigmoidanalytics.com> >> wrote: >>> >>> And this is how my classpath looks like >>> >>> Spark assembly has been built with Hive, including Datanucleus jars on >>> classpath >>> >>> ::/home/akhld/mobi/localcluster/spark-1/conf:/home/akhld/mobi/localcluster/spark-1/lib/spark-assembly-1.1.0-hadoop1.0.4.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-core-3.2.2.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-rdbms-3.2.1.jar:/home/akhld/mobi/localcluster/spark-1/lib/datanucleus-api-jdo-3.2.1.jar >>> >>> Thanks >>> Best Regards >>> >>> On Tue, Dec 16, 2014 at 10:33 PM, Daniel Haviv <danielru...@gmail.com> >>> wrote: >>>> >>>> Same here... >>>> # jar tf lib/spark-assembly-1.1.2-SNAPSHOT-hadoop2.3.0.jar | grep >>>> SparkSubmit.class >>>> *org/apache/spark/deploy/SparkSubmit.class* >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Dec 16, 2014 at 6:50 PM, Akhil Das <ak...@sigmoidanalytics.com> >>>> wrote: >>>>> >>>>> This is how it looks on my machine. >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> Thanks >>>>> Best Regards >>>>> >>>>> On Tue, Dec 16, 2014 at 9:33 PM, Daniel Haviv <danielru...@gmail.com> >>>>> wrote: >>>>>> >>>>>> That's the first thing I tried... still the same error: >>>>>> hdfs@ams-rsrv01:~$ export CLASSPATH=/tmp/spark/spark-branch-1.1/lib >>>>>> hdfs@ams-rsrv01:~$ cd /tmp/spark/spark-branch-1.1 >>>>>> hdfs@ams-rsrv01:/tmp/spark/spark-branch-1.1$ ./bin/spark-shell >>>>>> >>>>>> Spark assembly has been built with Hive, including Datanucleus jars >>>>>> on classpath >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>> org/apache/spark/deploy/SparkSubmit >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.spark.deploy.SparkSubmit >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >>>>>> at >>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:268) >>>>>> Could not find the main class: org.apache.spark.deploy.SparkSubmit. >>>>>> Program will exit. >>>>>> >>>>>> Thanks, >>>>>> Daniel >>>>>> >>>>>> On Tue, Dec 16, 2014 at 1:53 PM, Akhil Das < >>>>>> ak...@sigmoidanalytics.com> wrote: >>>>>>> >>>>>>> print the CLASSPATH and make sure the spark assembly jar is there in >>>>>>> the classpath >>>>>>> >>>>>>> Thanks >>>>>>> Best Regards >>>>>>> >>>>>>> On Tue, Dec 16, 2014 at 5:04 PM, Daniel Haviv <danielru...@gmail.com >>>>>>> > wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> I've built spark successfully with maven but when I try to run >>>>>>>> spark-shell I get the following errors: >>>>>>>> >>>>>>>> Spark assembly has been built with Hive, including Datanucleus jars >>>>>>>> on classpath >>>>>>>> >>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>>> org/apache/spark/deploy/SparkSubmit >>>>>>>> >>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>> org.apache.spark.deploy.SparkSubmit >>>>>>>> >>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>>> >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> >>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>> >>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >>>>>>>> >>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)at >>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:268)Could not >>>>>>>> find the main class: org.apache.spark.deploy.SparkSubmit. Program will >>>>>>>> exit. >>>>>>>> What can be wrong? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Daniel >>>>>>>> >>>>>>>
