I'd recommend you build a fill spark release with the new hadoop version; you should have built that locally earlier the same day (so that ivy/maven pick up the snapshot)
dev/make-distribution.sh -Pyarn,hadoop-2.7,hive -Dhadoop.version=2.9.0-SNAPSHOT; > On 18 Nov 2016, at 19:31, lminer <lmi...@hotmail.com> wrote: > > I'm trying to figure out how to run spark with a snapshot of Hadoop 2.8 that > I built myself. I'm unclear on the configuration needed to get spark to work > with the snapshot. > > I'm running spark on mesos. Per the spark documentation, I run spark-submit > as follows using the `spark-2.0.2-bin-without-hadoop`, but spark doesn't > appear to be finding hadoop 2.8. > > export SPARK_DIST_CLASSPATH=$(/path/to/hadoop2.8/bin/hadoop classpath) > spark-submit --verbose --master mesos://$MASTER_HOST/mesos > > I get the error: > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/fs/FSDataInputStream > at > org.apache.spark.deploy.SparkSubmitArguments.handle(SparkSubmitArguments.scala:403) > at > org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:163) > at > org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:98) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:117) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.fs.FSDataInputStream > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 5 more > > Any ideas on the proper configuration? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Run-spark-with-hadoop-snapshot-tp28105.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
