Try spark.yarn.user.classpath.first (see
https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN).
Also thread at
http://apache-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark-submit-td18030.html.
HTH,
Markus
On 02/03/2015 11:20 PM, Corey Nolet wrote:
In Hadoop 1.x TaskAttemptContext is a class (for example,
https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/TaskAttemptContext.html)
In Hadoop 2.x TaskAttemptContext is an interface
(https://hadoop.apache.org/docs/r2.4.0/api/org/apache/hadoop/mapreduce/TaskAttemptContext.html)
Short answer: if you downloaded spark-avro from the repo.maven.apache.org
repo you might be using an old version (pre-November 14, 2014) -
see timestamps at
http://repo.maven.apache.org/maven2/com/databricks/spark-avro_2.10/0.1/
Lots of changes at https://github.com/databricks/spark-avro since t
to it not being implemented (the job quits and says non
implemented method or something along those lines).
I will try going the spark shell and passing in the jar built from
github since I haven't tried that quite yet.
On Thu, Mar 12, 2015 at 6:44 PM, M. Dale <mailto:medal...@yaho
Tobias,
From http://spark.apache.org/docs/latest/configuration.html it seems
that there is an experimental property:
spark.files.userClassPathFirst
Whether to give user-added jars precedence over Spark's own jars when
loading classes in Executors. This feature can be used to mitigate
conf
I did not encounter this with my Avro records using Spark 1.10 (see
https://github.com/medale/spark-mail/blob/master/analytics/src/main/scala/com/uebercomputing/analytics/basic/UniqueSenderCounter.scala).
I do use the default Java serialization but all the fields in my Avro
object are Seriali
