Thanks Ted, I think I was able to resolve this issue without modifying spark by setting a relocation for jackson in my shaded jar.
On Tue, Jan 26, 2016 at 12:11 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Then maybe changing the following in pom.xml to 2.7.0 and rebuild Spark ? > > <fasterxml.jackson.version>2.5.3</fasterxml.jackson.version> > > On Tue, Jan 26, 2016 at 11:53 AM, asdf zxcv <benjamin.ha...@gmail.com> > wrote: > >> Hmm, this did not seem to resolve the issue. I also tried adding a >> relocation for jackson as well. >> >> On Tue, Jan 26, 2016 at 10:09 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> I wonder if the following change would solve the problem you described >>> (by shading jackson.core): >>> >>> diff --git a/pom.xml b/pom.xml >>> index fb77506..32a3237 100644 >>> --- a/pom.xml >>> +++ b/pom.xml >>> @@ -2177,6 +2177,7 @@ >>> <include>org.eclipse.jetty:jetty-util</include> >>> <include>org.eclipse.jetty:jetty-server</include> >>> <include>com.google.guava:guava</include> >>> + <include>com.fasterxml.jackson.core</include> >>> </includes> >>> </artifactSet> >>> <relocations> >>> >>> Cheers >>> >>> On Tue, Jan 26, 2016 at 9:34 AM, asdf zxcv <benjamin.ha...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm trying to run MaxMind GeoIP2 in a Spark task, but get a runtime >>>> error at init due to a NoSuchMethodError for ArrayNode from >>>> com.fasterxml.jackson.core:jackson-databind. >>>> This succeeds locally in unit tests, but fails in Spark tasks. >>>> >>>> I've excluded jackson-databind from all other dependencies, including >>>> Spark, since MaxMind is using the latest version of jackson-databind >>>> (2.7.0). >>>> >>>> `mvn dependency:tree` shows no conflicts for jackson-databind. There >>>> are numerous conflicts with other jackson utilities from org.codehaus, but >>>> this shouldn't be a problem since it is in a different namespace, right? >>>> >>>> I think this may be a problem with the shaded jar produced by >>>> maven-shade-plugin 2.4. Here is the config from pom.xml: >>>> pastebin.com/QzrhM5Ee >>>> >>>> `jar tvf shaded.jar` yields an entry for the missing class, >>>> com/fasterxml/jackson/databind/node/ArrayNode.class. >>>> META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml has >>>> the correct version, 2.7.0. The shaded jar expanded jackson-databind into >>>> its constituent *.class files instead of providing the jar itself; is this >>>> normal? >>>> >>>> My other thought is there is some issue with the classpath. Is it >>>> sufficient that the shaded jar contains the jackson-databind *.class files? >>>> How else can I investigate this? I know that GeoIP databse init succeeds >>>> locally with the jackson-databind jar in ~/.m2/ >>>> >>>> I've seen numerous issues with jackson-databind due to Spark using an >>>> old version and not shading this particular dependency, but this seems to >>>> have been resolved. I'm currently using version 1.3.1 and get the same >>>> result bumping the version to 1.6.0 >>>> >>>> More detail here: >>>> https://stackoverflow.com/questions/34958520/nosuchmethodexception-in-maxmind-geoip-dependency-jackson-databind-built-with-mv >>>> >>>> >>>> Thanks, >>>> >>>> Ben >>>> >>> >>> >> >
