Essentially, the class loader of Flink should be present in parent
hierarchy of context class loader. If FlinkUserCodeClassLoader doesn't
use context class loader, then it is actually impossible to use a
hierarchy like this:
system class loader -> application class loader -> user-defined class
loader (defines some UDFs to be used in flink program)
Flink now uses the application class loader, and so the UDFs fail to
deserialize on local flink, because the user-defined class loader is
bypassed. Moreover, there is no way to add additional classpath elements
for LocalEnvironment (as opposed to RemoteEnvironment). I'm able to hack
this by calling addURL method on the application class loader (which is
terribly hackish), but that works only on JDK <= 8. No sensible
workaround is available for JDK >= 9.
Alternative solution would be to enable adding jars to class loader when
using LocalEnvironment, but that looks a little odd.
Jan
On 9/2/19 11:02 AM, Aljoscha Krettek wrote:
Hi,
I actually don’t know whether that change would be ok. FlinkUserCodeClassLoader has
taken FlinkUserCodeClassLoader.class.getClassLoader() as the parent ClassLoader
before my change. See:
https://github.com/apache/flink/blob/release-1.2/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoader.java
<https://github.com/apache/flink/blob/release-1.2/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoader.java>.
I have the feeling that this might be on purpose because we want to have the
ClassLoader of the Flink Framework components be the parent ClassLoader, but I
could be wrong. Maybe Stephan would be most appropriate for answering this.
Best,
Aljoscha
On 30. Aug 2019, at 16:28, Till Rohrmann <trohrm...@apache.org> wrote:
Hi Jan,
this looks to me like a bug for which you could create a JIRA and PR to fix it.
Just to make sure, I've pulled in Aljoscha who is the author of this change to
check with him whether we are forgetting something.
Cheers,
Till
On Fri, Aug 30, 2019 at 3:44 PM Jan Lukavský <je...@seznam.cz
<mailto:je...@seznam.cz>> wrote:
Hi,
I have come across an issue with classloading in Flink's MiniCluster.
The issue is that when I run local flink job from a thread, that has a
non-default context classloader (for whatever reason), this classloader
is not taken into account when classloading user defined functions. This
is due to [1]. Is this behavior intentional, or can I file a JIRA and
use Thread.currentThread.getContextClassLoader() there? I have validated
that it fixes issues I'm facing.
Jan
[1]
https://github.com/apache/flink/blob/ce557839d762b5f1ec92aa1885fd3d2ae33d0d0b/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java#L280
<https://github.com/apache/flink/blob/ce557839d762b5f1ec92aa1885fd3d2ae33d0d0b/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java#L280>
