I believe you got to set following SPARK_HADOOP_VERSION=2.2.0 (or whatever your version is) SPARK_YARN=true
then type sbt/sbt assembly If you are using Maven to compile mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package Hope this helps -A On Fri, Apr 4, 2014 at 7:28 AM, Erik Freed <erikjfr...@codecision.com>wrote: > Hi All, > > I am not sure if this is a 0.9.0 problem to be fixed in 0.9.1 so perhaps > already being addressed, but I am having a devil of a time with a spark > 0.9.0 client jar for hadoop 2.X. If I go to the site and download: > > > - Download binaries for Hadoop 2 (HDP2, CDH5): find an Apache mirror > <http://www.apache.org/dyn/closer.cgi/incubator/spark/spark-0.9.0-incubating/spark-0.9.0-incubating-bin-hadoop2.tgz> > or direct file > download<http://d3kbcqa49mib13.cloudfront.net/spark-0.9.0-incubating-bin-hadoop2.tgz> > > I get a jar with what appears to be hadoop 1.0.4 that fails when using > hadoop 2.3.0. I have tried repeatedly to build the source tree with the > correct options per the documentation but always seemingly ending up with > hadoop 1.0.4. As far as I can tell the reason that the jar available on > the web site doesn't have the correct hadoop client in it, is because the > build itself is having that problem. > > I am about to try to troubleshoot the build but wanted to see if anyone > out there has encountered the same problem and/or if I am just doing > something dumb (!) > > > Anyone else using hadoop 2.X? How do you get the right client jar if so? > > cheers, > Erik > > -- > Erik James Freed > CoDecision Software > 510.859.3360 > erikjfr...@codecision.com > > 1480 Olympus Avenue > Berkeley, CA > 94708 > > 179 Maria Lane > Orcas, WA > 98245 >
