Hi Gordon, We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to pass -Phadoop-provided to Maven and avoid including Hadoop and its dependencies in the assembly jar.
-Sandy On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang <gw...@gopivotal.com> wrote: > In this page http://spark.apache.org/docs/0.9.0/running-on-yarn.html > > We have to use spark assembly to submit spark apps to yarn cluster. > And I checked the assembly jars of spark. It contains some yarn classes > which are added during compile time. The yarn classes are not what I want. > > My question is that is it possible to use other jars to submit spark app > to yarn cluster. > I do not want to use the assembly jar because it has yarn classes which > may overwrite the yarn class in HADOOP_CLASSPATH. If the yarn cluster is > upgraded, even if the YARN apis are same, spark has to be recompiled > against to the new version of yarn. > > Any help is appreciated ! Thanks. > > -- > Regards > Gordon Wang >
