Hi, I am new to spark. I would like to run jobs in Spark stand alone cluster mode.
No cluser managers other than psark is used.
I have tried wordcount from spark shell and stand alone scala app.
The code reads input from HDFS and writes the results to HDFS. uses 2 worker
nodes.
In shell the wordcount is successful, how ever my tries to run stand alone
programmes are in vain.
My environement
Ubuntu 12.04 - 32 bit
I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating
installed hadoop 2.2.0 as separate hduser and given permission to other users.
installed scala 2.10.3
installed sbt 0.13.1
Spark master act as HDFS master
I have one master and 2 worker nodes and HDFS is accessible in all nodes.
I downloaded example project and modified to use my spark cluster.
I started the sparkcluster at spark://192.168.0.138:7077
When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get following
error
gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run
[info] Loading project definition from
/home/gino/Test/spark-example-project/project
[info] Set current project to spark-example-project (in build
file:/home/gino/Test/spark-example-project/)
[info] Running com.Thinkpalm.spark.WordCountHDFS
[error] (run-main-0) java.lang.NoClassDefFoundError:
org/apache/spark/SparkContext
java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM
Could anyone give some pointers ... I have attached the project for reference.
Thanks and regards
Gino Mathews
spark-example-project.tgz
Description: spark-example-project.tgz
