One potential issue here is that mesos is using classifiers now to publish there jars. It might be that sbt-pack has trouble with dependencies that are published using classifiers. I'm pretty sure mesos is the only dependency in Spark that is using classifiers, so that's why I mention it.
On Sun, Jun 1, 2014 at 2:34 AM, Pierre B <pierre.borckm...@realimpactanalytics.com> wrote: > Hi all! > > We'be been using the sbt-pack sbt plugin > (https://github.com/xerial/sbt-pack) for building our standalone Spark > application for a while now. Until version 1.0.0, that worked nicely. > > For those who don't know the sbt-pack plugin, it basically copies all the > dependencies JARs from your local ivy/maven cache to a your target folder > (in target/pack/lib), and creates launch scripts (in target/pack/bin) for > your application (notably setting all these jars on the classpath). > > Now, since Spark 1.0.0 was released, we are encountering a weird error where > running our project with "sbt run" is fine but running our app with the > launch scripts generated by sbt-pack fails. > > After a (quite painful) investigation, it turns out some JARs are NOT copied > from the local ivy2 cache to the lib folder. I noticed that all the missing > jars contain "shaded" in their file name (but all not all jars with such > name are missing). > One of the missing JARs is explicitly from the Spark definition > (SparkBuild.scala, line 350): ``mesos-0.18.1-shaded-protobuf.jar``. > > This file is clearly present in my local ivy cache, but is not copied by > sbt-pack. > > Is there an evident reason for that? > > I don't know much about the shading mechanism, maybe I'm missing something > here? > > > Any help would be appreciated! > > Cheers > > Pierre > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Using-sbt-pack-with-Spark-1-0-0-tp6649.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
