[
https://issues.apache.org/jira/browse/FLINK-11205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887338#comment-16887338
]
Joey Echeverria commented on FLINK-11205:
-----------------------------------------
I'm seeing this same issue when running Flink jobs that are in a restart loop.
The restart policy is set with a very large number of max attempts and the
Flink job throws an exception on each record. I'll see if I can reproduce with
one of the example jobs.
> Task Manager Metaspace Memory Leak
> -----------------------------------
>
> Key: FLINK-11205
> URL: https://issues.apache.org/jira/browse/FLINK-11205
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.5.5, 1.6.2, 1.7.0
> Reporter: Nawaid Shamim
> Priority: Major
> Attachments: Screenshot 2018-12-18 at 12.14.11.png, Screenshot
> 2018-12-18 at 15.47.55.png
>
>
> Job Restarts causes task manager to dynamically load duplicate classes.
> Metaspace is unbounded and grows with every restart. YARN aggressively kill
> such containers but this affect is immediately seems on different task
> manager which results in death spiral.
> Task Manager uses dynamic loader as described in
> [https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html]
> {quote}
> *YARN*
> YARN classloading differs between single job deployments and sessions:
> * When submitting a Flink job/application directly to YARN (via {{bin/flink
> run -m yarn-cluster ...}}), dedicated TaskManagers and JobManagers are
> started for that job. Those JVMs have both Flink framework classes and user
> code classes in the Java classpath. That means that there is _no dynamic
> classloading_ involved in that case.
> * When starting a YARN session, the JobManagers and TaskManagers are started
> with the Flink framework classes in the classpath. The classes from all jobs
> that are submitted against the session are loaded dynamically.
> {quote}
> The above is not entirely true specially when you set {{-yD
> classloader.resolve-order=parent-first}} . We also above observed the above
> behaviour when submitting a Flink job/application directly to YARN (via
> {{bin/flink run -m yarn-cluster ...}}).
> !Screenshot 2018-12-18 at 12.14.11.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
