Hello everyone, I'm new to Flink and am encountering a nasty problem while trying to submit a streaming Flink Job. I'll try to explain it as thoroughly as possible.
Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as per documentation <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions> and I submit the job using yarn-session.sh and then flink run. Getting into more details, the Flink Job is a fat jar built with sbt assembly and the following dependencies: libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided" libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided" libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % flinkVersion % "provided" libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % "provided" libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion % "provided" libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0" libraryDependencies += "org.joda" % "joda-convert" % "1.8.3" libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2" assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false) inside the flink lib folder I have the following Jars: flink-connector-filesystem_2.10-1.3.2.jar flink-python_2.10-1.3.2.jar flink-connector-kinesis_2.10-1.3.2.jar flink-shaded-hadoop2-uber-1.3.2.jar flink-dist_2.10-1.3.2.jar log4j-1.2.17.jar flink-hbase_2.10-1.3.2.jar slf4j-log4j12-1.7.7.jar Right now, the issue is that when I flink run the job, I get NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it being inside the fat jar. So, I tried putting both hbase-common.jar and hbase-client.jars inside the lib folder, getting as a result another NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap. Now, I noticed that flink-connector-kinesis and all the hbase packages (both hbase-common and flink-hbase deps) have a different version of guava as dependency: the first one needs version 18.0, while the second one needs 12.0.1 version. I, then, tried to enable the shading of guava packages with the following: assemblyShadeRules in assembly := Seq( ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll ) Only to no avail, getting the exact same error as before. And I really don't know how to submit the job without pouring every single dependency inside the lib folder. I thought it could be a compatibility issue with the YARN cluster, but I tried launching the job locally only to get the same error. As an additional info, I get the following warnings if I don't specify flink packages as provided: [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1 [warn] +- org.typelevel:macro-compat_2.10:1.1.1 (depends on 2.1.0) [warn] +- org.apache.flink:flink-scala_2.10:1.3.2 (depends on 2.0.1) [warn] [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx [warn] +- org.apache.hbase:hbase-common:1.3.0 (depends on 6.1.26) [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 (depends on 6.1.26.hwx) [warn] [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom (depends on 3.8.0.Final) [warn] +- org.apache.zookeeper:zookeeper:3.4.6 (depends on 3.7.0.Final) [warn] [warn] * com.google.guava:guava:18.0 is selected over 12.0.1 [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 (depends on 18.0) [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 (depends on 12.0.1) [warn] +- org.apache.hbase:hbase-procedure:1.3.0 (depends on 12.0.1) [warn] +- org.apache.hbase:hbase-client:1.3.0 (depends on 12.0.1) [warn] +- org.apache.hbase:hbase-common:1.3.0 (depends on 12.0.1) [warn] +- org.apache.hbase:hbase-server:1.3.0 (depends on 12.0.1) [warn] [warn] * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2 (depends on 4.0.27.Final) [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 (depends on 4.0.27.Final) [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 (depends on 4.0.23.Final) [warn] +- org.apache.hbase:hbase-client:1.3.0 (depends on 4.0.23.Final) [warn] +- org.apache.hbase:hbase-server:1.3.0 (depends on 4.0.23.Final) [warn] [warn] * junit:junit:4.12 is selected over 3.8.1 [warn] +- org.apache.hbase:hbase-protocol:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-annotations:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-procedure:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-client:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-common:1.3.0 (depends on 4.12) [warn] +- org.apache.hbase:hbase-server:1.3.0 (depends on 4.12) [warn] +- jline:jline:0.9.94 (depends on 3.8.1) And here I am asking for help here. Thank you very much for your attention, Kind regards, Federico D'Ambrosio
