Ok Marcelo, Thanks for the quick and thorough replies. I’ll keep an eye on these tickets and the mailing list to see how things move along.
mn On Aug 20, 2014, at 1:33 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hi, > > On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell <matt.narr...@gmail.com> wrote: >> Specifying the driver-class-path yields behavior like >> https://issues.apache.org/jira/browse/SPARK-2420 and >> https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a >> can of worms here if I also need to replace the guava dependencies. > > There's a PR open for the Guava dependency. I'm not sure at the moment > if there will be a need to shade any other dependencies. > >> Wouldn’t calling “./make-distribution.sh —skip-java-test —hadoop 2.4.1 >> —with-yarn —tgz” include the appropriate versions of the hadoop libs into >> the spark jar? > > Yes. I was suggesting a workaround that just excludes any Hadoop > classes from your Spark jar; that way you're guaranteed to use > whatever Hadoop classes you have in your cluster. :-) > >> I’m trying to rebuild using the hadoop-provided profile, but I’m getting >> several build errors. Is this sufficient: mvm -Phadoop-provided clean >> package -Phadoop-2.4 -Pyarn -Dyarn.version=2.4.1 -Dhadoop.version=2.4.1 >> -DskipTests > > Hmm, I'm pretty sure that's worked for me in the past, but then I > haven't tried that in a while. > > -- > Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
