Hi Austin, Are you running your job detached in a per-job cluster? In that case inverted class loading does not work. This is because we add the user jar to the system class path, and there is no dynamic class loading involved at the moment [1].
You can try the YARN session mode, or – as Chesnay already suggested – shade the dependency on your side. Best, Gary [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Thanks Gary, > > I will try to look into why the child-first strategy seems to have failed > for this dependency. > > Best, > Austin > > On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <g...@ververica.com> wrote: > >> Hi, >> >> Actually Flink's inverted class loading feature was designed to mitigate >> problems with different versions of libraries that are not compatible with >> each other [1]. You may want to debug why it does not work for you. >> >> You can also try to use the Hadoop free Flink distribution, and export the >> HADOOP_CLASSPATH variable [2]. >> >> Best, >> Gary >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order >> [2] >> https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths >> >> On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> Hi, >>> >>> I recently experienced versioning clashes with the okio and okhttp when >>> trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After >>> investigating and talking to the okio team (see this issue) >>> <https://github.com/square/okio/issues/559>, I found that both okio and >>> okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, >>> respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded >>> jar. The okio team suggested that Flink should shade the uber jar to fix >>> the issue, but I'm wondering if there is something I can do on my end to >>> have all versions exist simultaneously. >>> >>> From the issue, here are the okio contents of the uber jar: >>> >>> *jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio* >>> >>> META-INF/maven/com.squareup.okio/ >>> META-INF/maven/com.squareup.okio/okio/ >>> META-INF/maven/com.squareup.okio/okio/pom.properties >>> META-INF/maven/com.squareup.okio/okio/pom.xml >>> okio/ >>> okio/AsyncTimeout$1.class >>> okio/AsyncTimeout$2.class >>> okio/AsyncTimeout$Watchdog.class >>> okio/AsyncTimeout.class >>> okio/Base64.class >>> okio/Buffer$1.class >>> okio/Buffer$2.class >>> okio/Buffer.class >>> okio/BufferedSink.class >>> okio/BufferedSource.class >>> okio/ByteString.class >>> okio/DeflaterSink.class >>> okio/ForwardingSink.class >>> okio/ForwardingSource.class >>> okio/ForwardingTimeout.class >>> okio/GzipSink.class >>> okio/GzipSource.class >>> okio/InflaterSource.class >>> okio/Okio$1.class >>> okio/Okio$2.class >>> okio/Okio$3.class >>> okio/Okio.class >>> okio/RealBufferedSink$1.class >>> okio/RealBufferedSink.class >>> okio/RealBufferedSource$1.class >>> okio/RealBufferedSource.class >>> okio/Segment.class >>> okio/SegmentPool.class >>> okio/SegmentedByteString.class >>> okio/Sink.class >>> okio/Source.class >>> okio/Timeout$1.class >>> okio/Timeout.class >>> okio/Util.class >>> >>> Thank you, >>> Austin Cawley-Edwards >>> >>
