Thanks I see. Do you guys have plan to port this to sbt?
On Wed, Apr 23, 2014 at 10:24 AM, Sandy Ryza <sandy.r...@cloudera.com>wrote: > Right, it only works for Maven > > > On Tue, Apr 22, 2014 at 6:23 PM, Gordon Wang <gw...@gopivotal.com> wrote: > >> Hi Sandy, >> >> Thanks for your reply ! >> >> Does this work for sbt ? >> >> I checked the commit, looks like only maven build has such option. >> >> >> >> On Wed, Apr 23, 2014 at 12:38 AM, Sandy Ryza <sandy.r...@cloudera.com>wrote: >> >>> Hi Gordon, >>> >>> We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to >>> pass -Phadoop-provided to Maven and avoid including Hadoop and its >>> dependencies in the assembly jar. >>> >>> -Sandy >>> >>> >>> On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang <gw...@gopivotal.com>wrote: >>> >>>> In this page http://spark.apache.org/docs/0.9.0/running-on-yarn.html >>>> >>>> We have to use spark assembly to submit spark apps to yarn cluster. >>>> And I checked the assembly jars of spark. It contains some yarn classes >>>> which are added during compile time. The yarn classes are not what I want. >>>> >>>> My question is that is it possible to use other jars to submit spark >>>> app to yarn cluster. >>>> I do not want to use the assembly jar because it has yarn classes which >>>> may overwrite the yarn class in HADOOP_CLASSPATH. If the yarn cluster is >>>> upgraded, even if the YARN apis are same, spark has to be recompiled >>>> against to the new version of yarn. >>>> >>>> Any help is appreciated ! Thanks. >>>> >>>> -- >>>> Regards >>>> Gordon Wang >>>> >>> >>> >> >> >> -- >> Regards >> Gordon Wang >> > > -- Regards Gordon Wang
