Thanks - that did solve my error, but instead got a different one:
java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/input/FileInputFormat
It seems like with that setting, spark can't find Hadoop.
On 7/7/14, Koert Kuipers wrote:
> spark has a setting to put user jars in front of
I don't have experience deploying to EC2. can you use add.jar conf to add
the missing jar at runtime ? I haven't tried this myself. Just a guess.
On Mon, Jul 7, 2014 at 12:16 PM, Chester Chen wrote:
> with "provided" scope, you need to provide the "provided" jars at the
> runtime yourself. I
with "provided" scope, you need to provide the "provided" jars at the
runtime yourself. I guess in this case Hadoop jar files.
On Mon, Jul 7, 2014 at 12:13 PM, Robert James
wrote:
> Thanks - that did solve my error, but instead got a different one:
> java.lang.NoClassDefFoundError:
> org/apac
Chester - I'm happy rebuilding Spark, but then how can I deploy it to EC2?
On 7/7/14, Chester Chen wrote:
> Have you tried to change the spark SBT scripts? You can change the
> dependency scope to "provided". This similar to compile scope, except JDK
> or container need to provide the dependenc
Have you tried to change the spark SBT scripts? You can change the
dependency scope to "provided". This similar to compile scope, except JDK
or container need to provide the dependency at runtime.
This assume the Spark will work with the new version of common libraries.
Of course, this is not a
spark has a setting to put user jars in front of classpath, which should do
the trick.
however i had no luck with this. see here:
https://issues.apache.org/jira/browse/SPARK-1863
On Mon, Jul 7, 2014 at 1:31 PM, Robert James wrote:
> spark-submit includes a spark-assembly uber jar, which has o
spark-submit includes a spark-assembly uber jar, which has older
versions of many common libraries. These conflict with some of the
dependencies we need. I have been racking my brain trying to find a
solution (including experimenting with ProGuard), but haven't been
able to: when we use spark-sub
