Hi, Probably multiple jars in the spark sharelib contain the jackson library and the system loads an incorrect one. If you turn on java verbose class loading, you can find which jar is loaded and you can clean up the sharelib.
Best, Sala On Sat, Apr 13, 2019 at 8:05 PM Lian Jiang <jiangok2...@gmail.com> wrote: > I tried spark.yarn.user.classpath.first = true and > oozie.launcher.mapreduce.user.classpath.first > = true but no luck. > > On Wed, Apr 10, 2019 at 11:47 PM Lian Jiang <jiangok2...@gmail.com> wrote: > > > Hi, > > > > I am using hortonwork HDP3.0 which has oozie 4.3.1 and spark 2.3.1. My > > spark job throws intermittent dependency error: > > > > 2019-04-11 06:29:07,178 [Driver] ERROR > > org.apache.spark.deploy.yarn.ApplicationMaster - User class threw > > exception: java.lang.NoClassDefFoundError: Could not initialize class > > com.fasterxml.jackson.databind.SerializationConfig > > > > java.lang.NoClassDefFoundError: Could not initialize class > > com.fasterxml.jackson.databind.SerializationConfig > > > > at > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:565) > > > > at > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:480) > > > > > > > > Caused by: com.fasterxml.jackson.databind.JsonMappingException: > > Incompatible Jackson version: 2.9.0 > > > > at > > > com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) > > > > at > > > com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) > > > > at > > > com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751) > > > > at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59) > > > > at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala) > > > > > > > > The same spark job running via spark-submit always pass. Why spark action > > has such issue? Thanks for any clue. > > >
