In spark 3.5.0 removed jackson-core-asl and jackson-mapper-asl <http://[SPARK-43225][BUILD][SQL] Remove jackson-core-asl and jackson-mapper-asl from pre-built distribution> those are with groupid org.codehaus.jackson.
Those others jackson-* are with groupid com.fasterxml.jackson.core tor. 2. nov. 2023 kl. 01:43 skrev eab...@163.com <eab...@163.com>: > Hi， > Please check the versions of jar files starting with "jackson-". Make > sure all versions are consistent. > jackson jar list in spark-3.3.0: > ================ > 2022/06/10 04:37 75,714 jackson-annotations-*2.13.3*.jar > 2022/06/10 04:37 374,895 jackson-core-*2.13.3*.jar > 2022/06/10 04:37 232,248 jackson-core-asl-1.9.13.jar > 2022/06/10 04:37 1,536,542 jackson-databind-*2.13.3*.jar > 2022/06/10 04:37 52,020 jackson-dataformat-yaml-*2.13.3*.jar > 2022/06/10 04:37 121,201 jackson-datatype-jsr310-*2.13.3*.jar > 2022/06/10 04:37 780,664 jackson-mapper-asl-1.9.13.jar > 2022/06/10 04:37 458,981 jackson-module-scala_2.12-*2.13.3*.jar > ================ > > Spark 3.3.0 uses Jackson version 2.13.3, while Spark 3.5.0 uses Jackson > version 2.15.2. > I think you can remove the lower version of Jackson package to keep the > versions consistent. > eabour > > > *From:* moshik.vi...@veeva.com.INVALID > *Date:* 2023-11-01 15:03 > *To:* user@spark.apache.org > *CC:* 'Saar Barhoom' <saar.barh...@veeva.com> > *Subject:* jackson-databind version mismatch > > Hi Spark team, > > > > On upgrading spark version from 3.2.1 to 3.4.1 got the following issue: > > *java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.JsonGenerator > com.fasterxml.jackson.databind.ObjectMapper.createGenerator(java.io.OutputStream, > com.fasterxml.jackson.core.JsonEncoding)'* > > * at > org.apache.spark.util.JsonProtocol$.toJsonString(JsonProtocol.scala:75)* > > * at > org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:74)* > > * at > org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:127)* > > * at scala.Option.map(Option.scala:230)* > > * at > org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)* > > * at > org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)* > > * at > org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)* > > * at > org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)* > > * at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)* > > * at > org.apache.spark.sql.Dataset.withAction(Dataset.scala:4165)* > > * at org.apache.spark.sql.Dataset.head(Dataset.scala:3161)* > > * at org.apache.spark.sql.Dataset.take(Dataset.scala:3382)* > > * at > org.apache.spark.sql.Dataset.takeAsList(Dataset.scala:3405)* > > * at > com.crossix.safemine.cloud.utils.DebugRDDLogger.showDataset(DebugRDDLogger.java:84)* > > * at > com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.getFillRateCountsWithSparkQuery(StatisticsTransformer.java:122)* > > * at > com.crossix.safemine.cloud.components.statistics.spark.StatisticsTransformer.calculateStatistics(StatisticsTransformer.java:61)* > > * at > com.crossix.safemine.cloud.components.statistics.spark.SparkFileStatistics.execute(SparkFileStatistics.java:102)* > > * at > com.crossix.safemine.cloud.StatisticsFlow.calculateAllStatistics(StatisticsFlow.java:146)* > > * at > com.crossix.safemine.cloud.StatisticsFlow.runStatistics(StatisticsFlow.java:119)* > > * at > com.crossix.safemine.cloud.StatisticsFlow.initialFileStatistics(StatisticsFlow.java:77)* > > * at > com.crossix.safemine.cloud.SMCFlow.process(SMCFlow.java:221)* > > * at > com.crossix.safemine.cloud.SMCFlow.execute(SMCFlow.java:132)* > > * at > com.crossix.safemine.cloud.SMCFlow.run(SMCFlow.java:91)* > > > > I see that that spark package contains the dependency: > > com.fasterxml.jackson.core:jackson-databind:jar:2.10.5:compile > > > > But jackson-databind 2.10.5 does not contain > *ObjectMapper.createGenerator(java.io.OutputStream, > com.fasterxml.jackson.core.JsonEncoding)* > > It was added on 2.11.0 > > > > Trying to upgrade jackson-databind fails with: > > *com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 > requires Jackson Databind version >= 2.10.0 and < 2.11.0* > > > > According to spark 3.3.0 release notes: "Upgrade Jackson to 2.13.3" but > spark package of 3.4.1 contains Jackson of 2.10.5 > > (https://spark.apache.org/releases/spark-release-3-3-0.html) > > What am I missing? > > > > -- > > *Moshik Vitas* > > Senior Software Developer, Crossix > > Veeva Systems > > m: +972-54-5326-400 > > moshik.vi...@veeva.com > > > > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
