Hello,
I still have problem after upgrading from flink 1.3.1 to 1.4.2
Our scenario looks like that:
we have container running on top of yarn. Machine that starts it has
installed flink and also loading some classpath libraries (e.g. hadoop) into
container.
there is seperate rest service that gets requests to run export job - it
uses YarnClusterClient and submitting Packaged program. Jars with actual
flink jobs are located in lib/ directory of service. On the machine where
Spring service is deployed we don't have flink installed.
For version 1.3 we had some libraries also loaded to container so that they
wont have to be loaded dynamically every time. If I understand it correctly
in strategy child-first it should not be needed any more, right?
Now our actual problems started to come up linked with class loading. After
restarting rest service first trigger of job is working fine, but next ones
are complaining on class versions that are loaded. We found out that
PackagedProgram is creating new classLoader on every creation. So we
overriden that behaviour so that we have static map holding one classloader
per jarFileName:
/private static final Map<String, ClassLoader> classLoaders =
Maps.newHashMap();
...
(constructor) {
...
classLoaders.computeIfAbsent(jarFile.getName(),
s -> getUserClassLoaderChildFirst(getAllLibraries(),
classPaths, getClass().getClassLoader()));
userCodeClassLoader = classLoaders.get(jarFile.getName());
this.mainClass = loadMainClass(entryPointClassName,
userCodeClassLoader);
}
/
I don't know if that is a good direction, but seems to solve an issue for
now. We are just not sure about stability of this solution - still tesing on
our internal environment but I'm affraid for now to proceed on production.
Can you give us any other things we could try out to deal with loading?
Also PackagedProgram is still using parentFirst strategy, in JobWithJars you
have method:
/
public static ClassLoader buildUserCodeClassLoader(List<URL> jars,
List<URL> classpaths, ClassLoader parent) {
URL[] urls = new URL[jars.size() + classpaths.size()];
for (int i = 0; i < jars.size(); i++) {
urls[i] = jars.get(i);
}
for (int i = 0; i < classpaths.size(); i++) {
urls[i + jars.size()] = classpaths.get(i);
}
return FlinkUserCodeClassLoaders.parentFirst(urls, parent);
}
/
is that correct to still point to parent?
In some of issues I found in mailing list, you suggest to set up container
to parent-first as a solving issue. We would like to find proper solution
working on supported child-first path and don't use workaround fix.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
