Hi Flavio,
when I encountered quite similar problem that you describe, it was
related to a static storage located in class that was loaded
"parent-first". In my case it was it was in java.lang.ClassValue, but it
might (and probably will be) different in your case. The problem is that
if user-code registers something in some (static) storage located in
class loaded with parent (TaskTracker) classloader, then its associated
classes will never be GC'd and Metaspace will grow. A good starting
point would be not to focus on biggest consumers of heap (in general),
but to look at where the 15k objects of type Class are referenced from.
That might help you figure this out. I'm not sure if there is something
that can be done in general to prevent this type of leaks. That would be
probably question on dev@ mailing list.
Best,
Jan
On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
Hello everybody,
I was writing this email when a similar thread on this mailing list
appeared..
The difference is that the other problem seems to be related
with Flink 1.10 on YARN and does not output anything helpful in
debugging the cause of the problem.
Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
session cluster (the job is submitted to the cluster using the CLI
client).
The problem arises when I submit the same job for about 20 times (this
number unfortunately is not deterministic and can change a little
bit). The error reported by the Task Executor is related to the ever
growing Metaspace..the error seems to be pretty detailed [1].
I found the same issue in some previous threads on this mailing list
and I've tried to figure it out the cause of the problem. The issue is
that looking at the objects allocated I don't really get an idea of
the source of the problem because the type of objects that are
consuming the memory are of general purpose (i.e. Bytes, Integers and
Strings)...these are my "top" memory consumers if looking at the
output of jmap -histo <PID>:
At run 0:
num #instances #bytes class name (module)
-------------------------------------------------------
1: 46238 13224056 [B (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
2: 3736 6536672 [I (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
3: 38081 913944 java.lang.String
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
4: 26 852384 [Lakka.dispatch.forkjoin.ForkJoinTask;
5: 7146 844984 java.lang.Class
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
At run 1:
1: 77.608 25.317.496 [B (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
2: 7.004 9.088.360 [I (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
3: 15.814 1.887.256 java.lang.Class
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
4: 67.381 1.617.144 java.lang.String
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
5: 3.906 1.422.960 [Ljava.util.HashMap$Node;
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
At run 6:
1: 81.408 25.375.400 [B (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
2: 12.479 7.249.392 [I (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
3: 29.090 3.496.168 java.lang.Class
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
4: 4.347 2.813.416 [Ljava.util.HashMap$Node;
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
5: 71.584 1.718.016 java.lang.String
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
At run 8:
1: 985.979 127.193.256 [B (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
2: 35.400 13.702.112 [I (java.base@11.0.9.1
<mailto:java.base@11.0.9.1>)
3: 260.387 6.249.288 java.lang.String
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
4: 148.836 5.953.440 java.util.HashMap$KeyIterator
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
5: 17.641 5.222.344 [Ljava.util.HashMap$Node;
(java.base@11.0.9.1 <mailto:java.base@11.0.9.1>)
Thanks in advance for any help,
Flavio
[1]
--------------------------------------------------------------------------------------------------
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
error has occurred. This can mean two things: either the job requires
a larger size of JVM metaspace to load classes or there is a class
loading leak. In the first case
'taskmanager.memory.jvm-metaspace.size' configuration option should be
increased. If the error persists (usually in cluster after several job
(re-)submissions) then there is probably a class loading leak in user
code or some of its dependencies which has to be investigated and
fixed. The task executor has to be shutdown...
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
~[?:?]
at
java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:?]
at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
~[?:?]
at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
~[flink-dist_2.12-1.11.0.jar:1.11.0]
at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
