Hi, I am using hortonwork HDP3.0 which has oozie 4.3.1 and spark 2.3.1. My spark job throws intermittent dependency error:
2019-04-11 06:29:07,178 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:565) at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:480) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751) at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59) at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala) The same spark job running via spark-submit always pass. Why spark action has such issue? Thanks for any clue.
