To avoid guava conflict, I use maven shade plugin to package my fat jar.
If you use maven, the shade plugin looks like this:
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>flink-job</shadedClassifierName>
<relocations>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>yourpackage.shaded.google</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>org/datanucleus/**</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</plugin>
...
To package fat jar:
mvn -e -DskipTests=true clean install shade:shade;
I hope, it helps.
- Kidong Lee.
2016-12-15 19:04 GMT+09:00 Yury Ruchin <yuri.ruc...@gmail.com>:
> Hi,
>
> I have run into a classpath issue when running Flink streaming job in YARN
> session. I package my app into a fat jar with all the dependencies needed.
> One of them is Google Guava. I then submit the jar to the session. The task
> managers pre-created by the session build their classpath from the
> FLINK_LIB_DIR and Hadoop / YARN lib directories. Unfortunately, there is a
> dated Guava version pulled along with Hadoop dependencies which conflicts
> with the version my app needs. Even worse, the Flink lib dir and Hadoop
> libraries take precedence over my jar.
>
> If I remember correctly, in Spark there is an option meaning "user
> classpath goes first when looking for classes". I couldn't find anything
> similar for Flink. I tried "flink run -C file:///path/to/my/libraries" in
> the hope to extend the classpath but the old Guava version takes precedence
> anyway.
>
> How else can I bump "priority" of my app jar during the load process?
>
> Thanks,
> Yury
>
