I see. The general known constraints on building your assembly jar for pyspark on Yarn are:
Java 6 NOT RedHat Maven Some of these are documented here <http://spark.apache.org/docs/latest/building-with-maven.html> (bottom). Maybe we should make it more explicit. 2014-11-13 2:31 GMT-08:00 jamborta <jambo...@gmail.com>: > it was built with 1.6 (tried 1.7, too) > > On Thu, Nov 13, 2014 at 2:52 AM, Andrew Or-2 [via Apache Spark User > List] <[hidden email] <http://user/SendEmail.jtp?type=node&node=18833&i=0>> > wrote: > > > Hey Jamborta, > > > > What java version did you build the jar with? > > > > 2014-11-12 16:48 GMT-08:00 jamborta <[hidden email]>: > >> > >> I have figured out that building the fat jar with sbt does not seem to > >> included the pyspark scripts using the following command: > >> > >> sbt/sbt -Pdeb -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive clean > >> publish-local assembly > >> > >> however the maven command works OK: > >> > >> mvn -Pdeb -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive -DskipTests > >> clean package > >> > >> am I running the correct sbt command? > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/No-module-named-pyspark-latest-built-tp18740p18787.html > >> Sent from the Apache Spark User List mailing list archive at > Nabble.com. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [hidden email] > >> For additional commands, e-mail: [hidden email] > >> > > > > > > > > ________________________________ > > If you reply to this email, your message will be added to the discussion > > below: > > > http://apache-spark-user-list.1001560.n3.nabble.com/No-module-named-pyspark-latest-built-tp18740p18797.html > > To unsubscribe from No module named pyspark - latest built, click here. > > NAML > > ------------------------------ > View this message in context: Re: No module named pyspark - latest built > <http://apache-spark-user-list.1001560.n3.nabble.com/No-module-named-pyspark-latest-built-tp18740p18833.html> > > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >
