Alex, Please find below the build.sbt. I am using the assembly command to
create a fat jar. These things were working fine until i changed the
version scala and spark cassandra connector and other dependent jars.
I tried to run my job on my local intellij then also i am getting the issue.
Thanks
Amit
name := """cft-common"""
ThisBuild / version := "0.0.1-SNAPSHOT"
ThisBuild/scalaVersion := "2.12.15"
Test / fork := true
Test / envVars := Map("env" -> "qa")
val jacksonVersion = "2.13.1"
val AkkaVersion = "2.6.17"
val sparkVersion = "3.1.0"
libraryDependencies += "com.typesafe" % "config" % "1.3.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.9.2"
//2021-10-29
Chuck replaced cake solutions with apache recent version
libraryDependencies += "com.typesafe.play" %% "play-json-joda" %
"2.9.2" ///2021-12-06
Chuck Updated play json to latest to match
libraryDependencies += "org.json4s" %% "json4s-native" % "3.6.12"
libraryDependencies += "com.datastax.cassandra" % "java-driver-core" %
"3.9.0"
libraryDependencies += "io.getquill" %% "quill-cassandra" % "3.4.9"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.2"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.8.1"
//2021-10-29
Chuck replaced cake solutions with apache recent version
libraryDependencies += "com.typesafe.akka" %% "akka-stream-kafka" %
"2.1.1" //2021-11-11
Chuck replaced cake solutions with akka recent version
libraryDependencies += "com.typesafe.akka" %% "akka-slf4j" %
AkkaVersion //2021-11-11
Chuck replaced cake solutions with akka recent version
libraryDependencies += "com.typesafe.akka" %% "akka-stream" %
AkkaVersion //2021-11-11
Chuck replaced cake solutions with akka recent version
libraryDependencies += "com.typesafe.akka" %% "akka-actor" %
AkkaVersion //2021-11-11
Chuck replaced cake solutions with akka recent version
libraryDependencies += "org.apache.logging.log4j" % "log4j-api" % "2.17.0"
libraryDependencies += "org.apache.logging.log4j" % "log4j-core" % "2.17.0"
/** pin jackson libraries to specific sbt version */
val jacksonDeps = Seq(
"com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion,
"com.fasterxml.jackson.core" % "jackson-annotations" % jacksonVersion,
"com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion,
"com.fasterxml.jackson.datatype" % "jackson-datatype-jdk8" % jacksonVersion,
"com.fasterxml.jackson.datatype" % "jackson-datatype-jsr310" %
jacksonVersion,
"com.fasterxml.jackson.module" %% "jackson-module-scala" % jacksonVersion,
"com.fasterxml.jackson.module" % "jackson-module-paranamer" % jacksonVersion
).filter(_.name != "jackson-annotations")
.map(_.exclude("com.fasterxml.jackson.core","jackson-annotations"))
def excludeJackson(fromDependency: ModuleID): ModuleID = {
jacksonDeps.foldLeft(fromDependency){ (libDep, jackSonDep) =>
libDep.exclude(jackSonDep.organization, jackSonDep.name)
}
}
// https://mvnrepository.com/artifact/com.google.guava/guava
//libraryDependencies += "com.google.guava" % "guava" % "16.0.1"
libraryDependencies ++= (
Seq(
"org.scalatest" %% "scalatest" % "3.0.1" % Test,
"org.scalacheck" %% "scalacheck" % "1.13.4" % Test)
).map(excludeJackson) ++
jacksonDeps
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind"
% "2.8.9"
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.8.9"
// https://mvnrepository.com/artifact/org.julienrf/play-json-derived-codecs
libraryDependencies += "org.julienrf" %% "play-json-derived-codecs" %
"5.0.0"
libraryDependencies += "org.scalatestplus.play" %%
"scalatestplus-play" % "3.0.0"
% Test
libraryDependencies += "net.liftweb" %% "lift-json" % "3.4.2"
libraryDependencies += "com.fasterxml.jackson.core" %
"jackson-annotations" % jacksonVersion force()
libraryDependencies += "net.liftweb" %% "lift-json" % "3.4.2"
val sparkDependencies = Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion ,
"org.apache.spark" %% "spark-catalyst" % sparkVersion ,
"org.apache.spark" %% "spark-streaming" % sparkVersion ,
"org.apache.spark" %% "spark-mllib" % sparkVersion ,
"com.datastax.spark" %% "spark-cassandra-connector" % "3.1.0", //
this includes cassandra-driver
"org.apache.spark" %% "spark-hive" % sparkVersion,
"org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion
//2021-11-11 updating streaming from .8 to .10 -- replaced cake
solutions with akka recent version
).map(_.exclude("org.slf4j","slf4j-api"))
.map(_.exclude("org.slf4j" , "jul-to-slf4j"))
.map(_.exclude("org.slf4j","jcl-over-slf4j"))
.map(_.exclude("org.slf4j","slf4j-log4j12"))
//.map(_.exclude("log4j","log4j"))
.map(_.exclude("org.apache.kafka", "kafka-clients"))
libraryDependencies ++=
sparkDependencies
.map(excludeJackson) ++
jacksonDeps
assembly / assemblyShadeRules := Seq(
ShadeRule.rename("com.google.**" -> "shade.com.google.@1"
).inAll
)
On Sat, Jan 22, 2022 at 7:55 AM Alex Ott <[email protected]> wrote:
> Show how do you execute your code - either you didn't pack it as uberjar,
> or didn't provide all necessary dependencies, if you're using `--jars`
> option. You may try `-assembly` variant when submitting your application
>
> Amit Sharma at "Fri, 21 Jan 2022 11:17:38 -0500" wrote:
> AS> Hello, I tried using a cassandra unshaded connector or normal
> connector both are giving the same error at runtime while
> AS> connecting to cassandra.
>
> AS> "com.datastax.spark" %% "spark-cassandra-connector-unshaded" % "2.4.2"
>
> AS> Or
>
> AS> "com.datastax.spark" %% "spark-cassandra-connector" % "3.1.0"
>
> AS> Russ similar issue is reported here also but no solution
>
> AS>
> https://community.datastax.com/questions/3519/issue-with-spring-boot-starter-data-cassandra-and.html
>
> AS> Caused by: java.lang.ClassNotFoundException:
> com.codahale.metrics.JmxReporter
> AS> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> AS> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> AS> at
> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
>
> AS> On Thu, Jan 20, 2022 at 5:17 PM Amit Sharma <[email protected]>
> wrote:
>
> AS> Hello, I am trying to upgrade my project from spark 2.3.3 to
> spark 3.2.0. While running the application locally I am getting
> AS> below error.
> AS>
> AS> Could you please let me know which version of the cassandra
> connector I should use. I am using below shaded connector but i
> AS> think that causing the issue
>
> AS> "com.datastax.spark" %% "spark-cassandra-connector-unshaded" %
> "2.4.2"
>
> AS> Caused by: java.lang.ClassNotFoundException:
> com.codahale.metrics.JmxReporter
> AS> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> AS> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> AS> at
> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
>
> AS> Thanks
> AS>
> AS> Amit
>
>
> --
> With best wishes, Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>
