with "provided" scope, you need to provide the "provided" jars at the runtime yourself. I guess in this case Hadoop jar files.
On Mon, Jul 7, 2014 at 12:13 PM, Robert James <srobertja...@gmail.com> wrote: > Thanks - that did solve my error, but instead got a different one: > java.lang.NoClassDefFoundError: > org/apache/hadoop/mapreduce/lib/input/FileInputFormat > > It seems like with that setting, spark can't find Hadoop. > > On 7/7/14, Koert Kuipers <ko...@tresata.com> wrote: > > spark has a setting to put user jars in front of classpath, which should > do > > the trick. > > however i had no luck with this. see here: > > > > https://issues.apache.org/jira/browse/SPARK-1863 > > > > > > > > On Mon, Jul 7, 2014 at 1:31 PM, Robert James <srobertja...@gmail.com> > > wrote: > > > >> spark-submit includes a spark-assembly uber jar, which has older > >> versions of many common libraries. These conflict with some of the > >> dependencies we need. I have been racking my brain trying to find a > >> solution (including experimenting with ProGuard), but haven't been > >> able to: when we use spark-submit, we get NoMethodErrors, even though > >> the code compiles fine, because the runtime classes are different than > >> the compile time classes! > >> > >> Can someone recommend a solution? We are using scala, sbt, and > >> sbt-assembly, but are happy using another tool (please provide > >> instructions how to). > >> > > >
