Victor Wong created FLINK-14626:
-----------------------------------
Summary: User jar packaged with hadoop dependencies may cause
class conflit with hadoop jars on yarn
Key: FLINK-14626
URL: https://issues.apache.org/jira/browse/FLINK-14626
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.9.1
Reporter: Victor Wong
Currently, the yarn application classpath is placed behind Flink classpath
(including user jars), which will cause conflict if the user jar accidentally
included hadoop dependencies.
{code:java}
// org.apache.flink.yarn.Utils#setupYarnClassPath
public static void setupYarnClassPath(Configuration conf, Map<String, String>
appMasterEnv) {
addToEnvironment(
appMasterEnv,
Environment.CLASSPATH.name(),
appMasterEnv.get(ENV_FLINK_CLASSPATH));
String[] applicationClassPathEntries = conf.getStrings(
YarnConfiguration.YARN_APPLICATION_CLASSPATH,
YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
for (String c : applicationClassPathEntries) {
addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
}
}
{code}
Maybe we should place the user jars behind yarn application classpath when
`org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is set
to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
