Please tale a look at SPARK-4048 and SPARK-5108 Cheers
On Sat, Jan 17, 2015 at 10:26 PM, Gil Vernik <g...@il.ibm.com> wrote: > Hi, > > I took a source code of Spark 1.2.0 and tried to build it together with > hadoop-openstack.jar ( To allow Spark an access to OpenStack Swift ) > I used Hadoop 2.6.0. > > The build was fine without problems, however in run time, while trying to > access "swift://" name space i got an exception: > java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass > at > > org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) > at > > org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732) > ...and the long stack trace goes here > > Digging into the problem i saw the following: > Jackson versions 1.9.X are not backward compatible, in particular they > removed JsonClass annotation. > Hadoop 2.6.0 uses jackson-asl version 1.9.13, while Spark has reference to > older version of jackson. > > This is the main pom.xml of Spark 1.2.0 : > > <dependency> > <!-- Matches the version of jackson-core-asl pulled in by avro --> > <groupId>org.codehaus.jackson</groupId> > <artifactId>jackson-mapper-asl</artifactId> > <version>1.8.8</version> > </dependency> > > Referencing 1.8.8 version, which is not compatible with Hadoop 2.6.0 . > If we change version to 1.9.13, than all will work fine and there will be > no run time exceptions while accessing Swift. The following change will > solve the problem: > > <dependency> > <!-- Matches the version of jackson-core-asl pulled in by avro --> > <groupId>org.codehaus.jackson</groupId> > <artifactId>jackson-mapper-asl</artifactId> > <version>1.9.13</version> > </dependency> > > I am trying to resolve this somehow so people will not get into this > issue. > Is there any particular need in Spark for jackson 1.8.8 and not 1.9.13? > Can we remove 1.8.8 and put 1.9.13 for Avro? > It looks to me that all works fine when Spark build with jackson 1.9.13, > but i am not an expert and not sure what should be tested. > > Thanks, > Gil Vernik. >
