anyhow i am ranting... sorry
On Wed, Feb 4, 2015 at 5:54 PM, Koert Kuipers wrote:
> yeah i think we have been lucky so far. but i dont really see how i have a
> choice. it would be fine if say hadoop exposes a very small set of
> libraries as part of the classpath. but if i look at the jars on h
yeah i think we have been lucky so far. but i dont really see how i have a
choice. it would be fine if say hadoop exposes a very small set of
libraries as part of the classpath. but if i look at the jars on hadoop
classpath its a ton! and why? why does parquet need to be included with
hadoop for ex
On Wed, Feb 4, 2015 at 1:12 PM, Koert Kuipers wrote:
> about putting stuff on classpath before spark or yarn... yeah you can shoot
> yourself in the foot with it, but since the container is isolated it should
> be ok, no? we have been using HADOOP_USER_CLASSPATH_FIRST forever with great
> success.
marcelo,
i was not aware of those fixes. its a fulltime job to keep up with spark...
i will take another look. it would be great if that works on spark
standalone also and resolves the issues i experienced before.
about putting stuff on classpath before spark or yarn... yeah you can shoot
yourself
My mistake Marcello, I was looking at the wrong message. That reply was
meant for bo yang.
On Feb 4, 2015 4:02 PM, "Marcelo Vanzin" wrote:
> Hi Corey,
>
> On Wed, Feb 4, 2015 at 12:44 PM, Corey Nolet wrote:
> >> Another suggestion is to build Spark by yourself.
> >
> > I'm having trouble seeing
Hi Corey,
On Wed, Feb 4, 2015 at 12:44 PM, Corey Nolet wrote:
>> Another suggestion is to build Spark by yourself.
>
> I'm having trouble seeing what you mean here, Marcelo. Guava is already
> shaded to a different package for the 1.2.0 release. It shouldn't be causing
> conflicts.
That wasn't m
Hi Koert,
On Wed, Feb 4, 2015 at 11:35 AM, Koert Kuipers wrote:
> do i understand it correctly that on yarn the the customer jars are truly
> placed before the yarn and spark jars on classpath? meaning at container
> construction time, on the same classloader? that would be great news for me.
> i
i have never been a big fan of shading, since it leads to the same library
being packaged many times. for example, its not unusual to have ASM 10
times in a jar because of the shading policy they promote. and all that
because they broke one signature and without a really good reason.
i am a big fa
> leading to a real classloader hell if you tried to add a newer version of
jar that spark already used.
Spark at least makes this easier on the Guava side [1] by shading the
package names internally so that there's no possibility of a conflict.
Elasticsearch & Storm do this too for many of their
the whole spark.files.userClassPathFirs never really worked for me in
standalone mode, since jars were added dynamically which means they had
different classloaders leading to a real classloader hell if you tried to
add a newer version of jar that spark already used. see:
https://issues.apache.org/
Hi Corey,
When you run on Yarn, Yarn's libraries are placed in the classpath,
and they have precedence over your app's. So, with Spark 1.2, you'll
get Guava 11 in your classpath (with Spark 1.1 and earlier you'd get
Guava 14 from Spark, so still a problem for you).
Right now, the option Markus me
Hi Corey,
I see. Thanks for making it clear. I may be lucky not hitting code path of
such Guava classes. But I hit some other jar conflicts when using Spark to
connect to AWS S3. Then I had to manually try each version of
org.apache.httpcomponents until I found a proper old version.
Another sugge
Bo yang-
I am using Spark 1.2.0 and undoubtedly there are older Guava classes which
are being picked up and serialized with the closures when they are sent
from the driver to the executors because the class serial version ids don't
match from the driver to the executors. Have you tried doing this?
Corey,
Which version of Spark do you use? I am using Spark 1.2.0, and guava 15.0.
It seems fine.
Best,
Bo
On Tue, Feb 3, 2015 at 8:56 PM, M. Dale wrote:
> Try spark.yarn.user.classpath.first (see
> https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN).
> Also thread at
> h
Try spark.yarn.user.classpath.first (see
https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN).
Also thread at
http://apache-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark-submit-td18030.html.
HTH,
Markus
On 02/03/2015 11:20 PM, Corey Nolet wrote:
15 matches
Mail list logo
