There is no (accurate) reference of included dependencies for the
flink-shaded-hadoop uber jars. The contained NOTICE file is a good
starting point, but for the time being we're using a generalized version
that we apply to all hadoop versions (so some things may be missing).
I believe for hadoop 2.8.4 the following dependencies are contained,
based on the output of the maven-shade-plugin:
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google.code.gson:gson:2.2.4
com.google.guava:guava:11.0.2
com.google.guava:guava:11.0.2
com.google.protobuf:protobuf-java:2.5.0
com.google.protobuf:protobuf-java:2.5.0
com.jamesmurty.utils:java-xmlbuilder:0.4
com.jcraft:jsch:0.1.54
com.nimbusds:nimbus-jose-jwt:4.41.1
com.squareup.okhttp:okhttp:2.4.0
com.squareup.okio:okio:1.4.0
com.sun.jersey:jersey-client:1.9
com.thoughtworks.paranamer:paranamer:2.7
commons-beanutils:commons-beanutils:1.9.3
commons-cli:commons-cli:1.3.1
commons-codec:commons-codec:1.10
commons-collections:commons-collections:3.2.2
commons-configuration:commons-configuration:1.7
commons-daemon:commons-daemon:1.0.13
commons-digester:commons-digester:1.8.1
commons-io:commons-io:2.4
commons-lang:commons-lang:2.6
commons-logging:commons-logging:1.1.3
commons-net:commons-net:3.1
io.netty:netty-all:4.0.23.Final
io.netty:netty-all:4.0.23.Final
io.netty:netty:3.6.2.Final
io.netty:netty:3.6.2.Final
javax.activation:activation:1.1
javax.servlet:servlet-api:2.5
javax.xml.bind:jaxb-api:2.2.2
javax.xml.stream:stax-api:1.0-2
net.java.dev.jets3t:jets3t:0.9.0
net.java.dev.jets3t:jets3t:0.9.0
net.minidev:accessors-smart:1.2
net.minidev:json-smart:2.3
org.apache.avro:avro:1.8.2
org.apache.commons:commons-compress:1.18
org.apache.commons:commons-math3:3.5
org.apache.curator:curator-client:2.7.1
org.apache.curator:curator-client:2.7.1
org.apache.curator:curator-framework:2.7.1
org.apache.curator:curator-framework:2.7.1
org.apache.curator:curator-recipes:2.7.1
org.apache.curator:curator-recipes:2.7.1
org.apache.directory.api:api-asn1-api:1.0.0-M20
org.apache.directory.api:api-util:1.0.0-M20
org.apache.directory.server:apacheds-i18n:2.0.0-M15
org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
org.apache.hadoop:hadoop-annotations:2.8.4
org.apache.hadoop:hadoop-annotations:2.8.4
org.apache.hadoop:hadoop-auth:2.8.4
org.apache.hadoop:hadoop-auth:2.8.4
org.apache.hadoop:hadoop-common:2.8.4
org.apache.hadoop:hadoop-common:2.8.4
org.apache.hadoop:hadoop-hdfs-client:2.8.4
org.apache.hadoop:hadoop-hdfs-client:2.8.4
org.apache.hadoop:hadoop-hdfs:2.8.4
org.apache.hadoop:hadoop-hdfs:2.8.4
org.apache.hadoop:hadoop-mapreduce-client-core:2.8.4
org.apache.hadoop:hadoop-mapreduce-client-core:2.8.4
org.apache.hadoop:hadoop-yarn-api:2.8.4
org.apache.hadoop:hadoop-yarn-api:2.8.4
org.apache.hadoop:hadoop-yarn-client:2.8.4
org.apache.hadoop:hadoop-yarn-client:2.8.4
org.apache.hadoop:hadoop-yarn-common:2.8.4
org.apache.hadoop:hadoop-yarn-common:2.8.4
org.apache.htrace:htrace-core4:4.0.1-incubating
org.apache.httpcomponents:httpclient:4.5.3
org.apache.httpcomponents:httpclient:4.5.3
org.apache.httpcomponents:httpcore:4.4.6
org.apache.httpcomponents:httpcore:4.4.6
org.apache.zookeeper:zookeeper:3.4.10
org.codehaus.jackson:jackson-core-asl:1.9.13
org.codehaus.jackson:jackson-core-asl:1.9.13
org.codehaus.jackson:jackson-jaxrs:1.9.13
org.codehaus.jackson:jackson-jaxrs:1.9.13
org.codehaus.jackson:jackson-mapper-asl:1.9.13
org.codehaus.jackson:jackson-mapper-asl:1.9.13
org.codehaus.jackson:jackson-xc:1.9.13
org.codehaus.jackson:jackson-xc:1.9.13
org.fusesource.leveldbjni:leveldbjni-all:1.8
org.mortbay.jetty:jetty-sslengine:6.1.26
org.tukaani:xz:1.5
org.xerial.snappy:snappy-java:1.1.4
xerces:xercesImpl:2.9.1
xml-apis:xml-apis:1.3.04
xmlenc:xmlenc:0.52
On 28.02.2019 15:54, Austin Cawley-Edwards wrote:
Hi Gary,
No, I am running a YARN session (which I start
with: flink-yarn-session --slots 4 --taskManagerMemory 16GB
--jobManagerMemory 3GB --detached) and submit jobs through the REST
interface. Thank you for the tips - I will probably shade it on my
side. Is there an official location that the uber jar dependencies are
documented that I can reference for future dependency additions?
Best,
Austin
On Thu, Feb 28, 2019 at 7:13 AM Gary Yao <g...@ververica.com
<mailto:g...@ververica.com>> wrote:
Hi Austin,
Are you running your job detached in a per-job cluster? In that
case inverted
class loading does not work. This is because we add the user jar
to the system
class path, and there is no dynamic class loading involved at the
moment [1].
You can try the YARN session mode, or – as Chesnay already
suggested – shade
the dependency on your side.
Best,
Gary
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink
On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards
<austin.caw...@gmail.com <mailto:austin.caw...@gmail.com>> wrote:
Thanks Gary,
I will try to look into why the child-first strategy seems to
have failed for this dependency.
Best,
Austin
On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <g...@ververica.com
<mailto:g...@ververica.com>> wrote:
Hi,
Actually Flink's inverted class loading feature was
designed to mitigate
problems with different versions of libraries that are not
compatible with
each other [1]. You may want to debug why it does not work
for you.
You can also try to use the Hadoop free Flink
distribution, and export the
HADOOP_CLASSPATH variable [2].
Best,
Gary
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths
On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards
<austin.caw...@gmail.com <mailto:austin.caw...@gmail.com>>
wrote:
Hi,
I recently experienced versioning clashes with the
okio and okhttp when trying to deploy a Flink 1.6.0
app to AWS EMR on Hadoop 2.8.4. After investigating
and talking to the okio team (see this issue)
<https://github.com/square/okio/issues/559>, I found
that both okio and okhttp exist in the Flink uber jar
with versions 1.4.0 and 2.4.0, respectively, whereas
I'm including versions 2.2.2 and 3.13.1 in my shaded
jar. The okio team suggested that Flink should shade
the uber jar to fix the issue, but I'm wondering if
there is something I can do on my end to have all
versions exist simultaneously.
From the issue, here are the okio contents of the uber
jar:
*jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio*
META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class
Thank you,
Austin Cawley-Edwards
