Hi,

why do you need to add hbase-common as a separate dependency? Doesn't the
"flink-hbase" dependency transitively pull in hbase?

On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> If Guava 18.0 is used to build hbase 1.3, there would be compilation
> errors such as the following:
>
> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/
> hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25]
> error: cannot find symbol
> [ERROR]   symbol:   method stopAndWait()
> [ERROR]   location: variable replicationEndpoint of type
> ReplicationEndpoint
> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/
> hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47]
> error: cannot find symbol
> [ERROR]   symbol:   method start()
> [ERROR]   location: variable replicationEndpoint of type
> ReplicationEndpoint
>
> Maybe you can shade the guava dependency in hbase 1.3
>
> In the upcoming hbase 2.0 release, third party dependencies such as guava
> and netty are shaded. Meaning there wouldn't be such conflict with other
> components you may use.
>
> On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio <
> federico.dambro...@smartlab.ws> wrote:
>
>> Hello everyone, I'm new to Flink and am encountering a nasty problem
>> while trying to submit a streaming Flink Job. I'll try to explain it as
>> thoroughly as possible.
>>
>> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
>> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
>> per documentation
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
>> and I submit the job using yarn-session.sh and then flink run.
>>
>> Getting into more details, the Flink Job is a fat jar built with sbt
>> assembly and the following dependencies:
>>
>> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % 
>> "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % 
>> flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % 
>> flinkVersion % "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % 
>> "provided"
>> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % 
>> flinkVersion % "provided"
>> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
>> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
>> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"
>>
>>
>> assemblyOption in assembly := (assemblyOption in 
>> assembly).value.copy(includeScala = false)
>>
>>
>> inside the flink lib folder I have the following Jars:
>>
>> flink-connector-filesystem_2.10-1.3.2.jar
>> flink-python_2.10-1.3.2.jar
>> flink-connector-kinesis_2.10-1.3.2.jar
>> flink-shaded-hadoop2-uber-1.3.2.jar
>> flink-dist_2.10-1.3.2.jar
>> log4j-1.2.17.jar
>> flink-hbase_2.10-1.3.2.jar
>> slf4j-log4j12-1.7.7.jar
>>
>> Right now, the issue is that when I flink run the job, I get
>> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it
>> being inside the fat jar. So, I tried putting both hbase-common.jar and
>> hbase-client.jars inside the lib folder, getting as a result another
>> NoClassDefFound, only, this time, for com/google/common/collect/List
>> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase
>> packages (both hbase-common and flink-hbase deps) have a different version
>> of guava as dependency: the first one needs version 18.0, while the second
>> one needs 12.0.1 version. I, then, tried to enable the shading of guava
>> packages with the following:
>>
>> assemblyShadeRules in assembly := Seq(
>>   ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
>> )
>>
>>
>> Only to no avail, getting the exact same error as before. And I really
>> don't know how to submit the job without pouring every single dependency
>> inside the lib folder. I thought it could be a compatibility issue with the
>> YARN cluster, but I tried launching the job locally only to get the same
>> error.
>>
>> As an additional info, I get the following warnings if I don't specify
>> flink packages as provided:
>>
>> [warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over
>> 2.0.1
>> [warn]         +- org.typelevel:macro-compat_2.10:1.1.1
>> (depends on 2.1.0)
>> [warn]         +- org.apache.flink:flink-scala_2.10:1.3.2
>> (depends on 2.0.1)
>> [warn]
>> [warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over
>> 6.1.26.hwx
>> [warn]         +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 6.1.26)
>> [warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>> (depends on 6.1.26.hwx)
>> [warn]
>> [warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
>> [warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom
>> (depends on 3.8.0.Final)
>> [warn]         +- org.apache.zookeeper:zookeeper:3.4.6
>> (depends on 3.7.0.Final)
>> [warn]
>> [warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
>> [warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
>> (depends on 18.0)
>> [warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 12.0.1)
>> [warn]         +- org.apache.hbase:hbase-procedure:1.3.0
>> (depends on 12.0.1)
>> [warn]         +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 12.0.1)
>> [warn]         +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 12.0.1)
>> [warn]         +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 12.0.1)
>> [warn]
>> [warn]     * io.netty:netty-all:4.0.27.Final is selected over
>> 4.0.23.Final
>> [warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2
>> (depends on 4.0.27.Final)
>> [warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2
>> (depends on 4.0.27.Final)
>> [warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 4.0.23.Final)
>> [warn]         +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 4.0.23.Final)
>> [warn]         +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 4.0.23.Final)
>> [warn]
>> [warn]     * junit:junit:4.12 is selected over 3.8.1
>> [warn]         +- org.apache.hbase:hbase-protocol:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-annotations:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-procedure:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-client:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-common:1.3.0
>> (depends on 4.12)
>> [warn]         +- org.apache.hbase:hbase-server:1.3.0
>> (depends on 4.12)
>> [warn]         +- jline:jline:0.9.94
>> (depends on 3.8.1)
>>
>> And here I am asking for help here.
>>
>> Thank you very much for your attention,
>> Kind regards,
>>
>> Federico D'Ambrosio
>>
>
>

Reply via email to