FYI I filed SPARK-3647 to track the fix (some people internally have
bumped into this also).
On Mon, Sep 22, 2014 at 1:28 PM, Cody Koeninger wrote:
> We've worked around it for the meantime by excluding guava from transitive
> dependencies in the job assembly and specifying the same version of gu
We've worked around it for the meantime by excluding guava from transitive
dependencies in the job assembly and specifying the same version of guava
14 that spark is using. Obviously things break whenever a guava 15 / 16
feature is used at runtime, so a long term solution is needed.
On Mon, Sep 2
Hmmm, a quick look at the code indicates this should work for
executors, but not for the driver... (maybe this deserves a bug being
filed, if there isn't one already?)
If it's feasible for you, you could remove the Optional.class file
from the Spark assembly you're using.
On Mon, Sep 22, 2014 at
We're using Mesos, is there a reasonable expectation that
spark.files.userClassPathFirst will actually work?
On Mon, Sep 22, 2014 at 1:42 PM, Marcelo Vanzin wrote:
> Hi Cody,
>
> I'm still writing a test to make sure I understood exactly what's
> going on here, but from looking at the stack trac
Hi Cody,
I'm still writing a test to make sure I understood exactly what's
going on here, but from looking at the stack trace, it seems like the
newer Guava library is picking up the "Optional" class from the Spark
assembly.
Could you try one of the options that put the user's classpath before
th
Hi Marcelo,
Interested to hear the approach to be taken. Shading guava itself seems
extreme, but that might make sense.
Gary
On Sat, Sep 20, 2014 at 9:38 PM, Marcelo Vanzin wrote:
> Hmm, looks like the hack to maintain backwards compatibility in the
> Java API didn't work that well. I'll take
Hmm, looks like the hack to maintain backwards compatibility in the
Java API didn't work that well. I'll take a closer look at this when I
get to work on Monday.
On Fri, Sep 19, 2014 at 10:30 PM, Cody Koeninger wrote:
> After the recent spark project changes to guava shading, I'm seeing issues
>
After the recent spark project changes to guava shading, I'm seeing issues
with the datastax spark cassandra connector (which depends on guava 15.0)
and the datastax cql driver (which depends on guava 16.0.1)
Building an assembly for a job (with spark marked as provided) that
includes either guava
