Hi Flink Community,

We are building on top of  *org.apache.flink.yarn.YarnClusterDescriptor *to
submit a flink application from Java code to YARN cluster, in the
application mode. We are setting the classpath as the value of *the
yarn.provided.lib.dirs
*property under the yarn configuration.

By playing with the YarnClusterDescriptor code I have two questions that I
hope to get some answers:
1. YarnClusterDescriptor seems to force the classpath loading in
alphabetical order. See code here
<https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L966>.
Is there any specific reason for doing that? If I'd like to enforce my own
order is it possible now?
2. Looks like the *flink-dist.jar* is treated separately from the other
classpath classes. In the *YarnApplicationFileUploader* class,
the registerMultipleLocalResources method will skip the jar if it is a dist
jar. See the code here
<https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java#L283>.
With the current behavior it seems it will always place the flink-dist.jar
at the end of the classpath. Is there any reason that Flink wants to treat
the *flink-dist.jar* separately from other jars?

In our classpath loading we are hoping to enforce certain order because
different jars may contain the same dependent library but with different
versions. We hope to force the order so that we can load the correct
library version as we want.


Thanks
Leon

Reply via email to