Hi, why do you need to add hbase-common as a separate dependency? Doesn't the "flink-hbase" dependency transitively pull in hbase?
On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yuzhih...@gmail.com> wrote: > If Guava 18.0 is used to build hbase 1.3, there would be compilation > errors such as the following: > > [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/ > hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] > error: cannot find symbol > [ERROR] symbol: method stopAndWait() > [ERROR] location: variable replicationEndpoint of type > ReplicationEndpoint > [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/ > hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] > error: cannot find symbol > [ERROR] symbol: method start() > [ERROR] location: variable replicationEndpoint of type > ReplicationEndpoint > > Maybe you can shade the guava dependency in hbase 1.3 > > In the upcoming hbase 2.0 release, third party dependencies such as guava > and netty are shaded. Meaning there wouldn't be such conflict with other > components you may use. > > On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio < > federico.dambro...@smartlab.ws> wrote: > >> Hello everyone, I'm new to Flink and am encountering a nasty problem >> while trying to submit a streaming Flink Job. I'll try to explain it as >> thoroughly as possible. >> >> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version >> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as >> per documentation >> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions> >> and I submit the job using yarn-session.sh and then flink run. >> >> Getting into more details, the Flink Job is a fat jar built with sbt >> assembly and the following dependencies: >> >> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % >> "provided" >> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % >> flinkVersion % "provided" >> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % >> flinkVersion % "provided" >> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % >> "provided" >> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % >> flinkVersion % "provided" >> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0" >> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3" >> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2" >> >> >> assemblyOption in assembly := (assemblyOption in >> assembly).value.copy(includeScala = false) >> >> >> inside the flink lib folder I have the following Jars: >> >> flink-connector-filesystem_2.10-1.3.2.jar >> flink-python_2.10-1.3.2.jar >> flink-connector-kinesis_2.10-1.3.2.jar >> flink-shaded-hadoop2-uber-1.3.2.jar >> flink-dist_2.10-1.3.2.jar >> log4j-1.2.17.jar >> flink-hbase_2.10-1.3.2.jar >> slf4j-log4j12-1.7.7.jar >> >> Right now, the issue is that when I flink run the job, I get >> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it >> being inside the fat jar. So, I tried putting both hbase-common.jar and >> hbase-client.jars inside the lib folder, getting as a result another >> NoClassDefFound, only, this time, for com/google/common/collect/List >> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase >> packages (both hbase-common and flink-hbase deps) have a different version >> of guava as dependency: the first one needs version 18.0, while the second >> one needs 12.0.1 version. I, then, tried to enable the shading of guava >> packages with the following: >> >> assemblyShadeRules in assembly := Seq( >> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll >> ) >> >> >> Only to no avail, getting the exact same error as before. And I really >> don't know how to submit the job without pouring every single dependency >> inside the lib folder. I thought it could be a compatibility issue with the >> YARN cluster, but I tried launching the job locally only to get the same >> error. >> >> As an additional info, I get the following warnings if I don't specify >> flink packages as provided: >> >> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over >> 2.0.1 >> [warn] +- org.typelevel:macro-compat_2.10:1.1.1 >> (depends on 2.1.0) >> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2 >> (depends on 2.0.1) >> [warn] >> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over >> 6.1.26.hwx >> [warn] +- org.apache.hbase:hbase-common:1.3.0 >> (depends on 6.1.26) >> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 >> (depends on 6.1.26.hwx) >> [warn] >> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final >> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom >> (depends on 3.8.0.Final) >> [warn] +- org.apache.zookeeper:zookeeper:3.4.6 >> (depends on 3.7.0.Final) >> [warn] >> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1 >> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 >> (depends on 18.0) >> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >> (depends on 12.0.1) >> [warn] +- org.apache.hbase:hbase-procedure:1.3.0 >> (depends on 12.0.1) >> [warn] +- org.apache.hbase:hbase-client:1.3.0 >> (depends on 12.0.1) >> [warn] +- org.apache.hbase:hbase-common:1.3.0 >> (depends on 12.0.1) >> [warn] +- org.apache.hbase:hbase-server:1.3.0 >> (depends on 12.0.1) >> [warn] >> [warn] * io.netty:netty-all:4.0.27.Final is selected over >> 4.0.23.Final >> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2 >> (depends on 4.0.27.Final) >> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 >> (depends on 4.0.27.Final) >> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >> (depends on 4.0.23.Final) >> [warn] +- org.apache.hbase:hbase-client:1.3.0 >> (depends on 4.0.23.Final) >> [warn] +- org.apache.hbase:hbase-server:1.3.0 >> (depends on 4.0.23.Final) >> [warn] >> [warn] * junit:junit:4.12 is selected over 3.8.1 >> [warn] +- org.apache.hbase:hbase-protocol:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-annotations:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-procedure:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-client:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-common:1.3.0 >> (depends on 4.12) >> [warn] +- org.apache.hbase:hbase-server:1.3.0 >> (depends on 4.12) >> [warn] +- jline:jline:0.9.94 >> (depends on 3.8.1) >> >> And here I am asking for help here. >> >> Thank you very much for your attention, >> Kind regards, >> >> Federico D'Ambrosio >> > >
