In regards to the metaspace memory issue, I was able to get a heap dump and the following is the output:
Problem Suspect 1 One instance of *"java.lang.ref.Finalizer"* loaded by *"<system class loader>"* occupies *4,112,624 (11.67%)* bytes. The instance is referenced by *sun.misc.Cleaner @ 0xb5d6b520* , loaded by *"<system class loader>"*. The memory is accumulated in one instance of *"java.lang.Object[]"* loaded by *"<system class loader>"*. Problem Suspect 2 33 instances of *"org.apache.flink.util.ChildFirstClassLoader"*, loaded by *"sun.misc.Launcher$AppClassLoader @ 0xb4068680"* occupy *6,615,416 (18.76%)*bytes. Based on this, I'm not clear on what needs to be done to solve this. On Tue, Sep 22, 2020 at 3:10 PM Claude M <claudemur...@gmail.com> wrote: > Thanks for your responses. > 1. There were no job re-starts prior to the metaspace OEM. > 2. I tried increasing the CPU request and still encountered the problem. > Any configuration change I make to the job manager, whether it's in the > flink-conf.yaml or increasing the pod's CPU/memory request, results > with this problem. > > > On Tue, Sep 22, 2020 at 12:04 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> Thanks for the input, Brain. >> >> This looks like what we are looking for. The issue is fixed in 1.10.3, >> which also matches this problem occurred in 1.10.2. >> >> Maybe Claude can further confirm it. >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Tue, Sep 22, 2020 at 10:57 AM Zhou, Brian <b.z...@dell.com> wrote: >> >>> Hi Xintong and Claude, >>> >>> >>> >>> In our internal tests, we also encounter these two issues and we spent >>> much time debugging them. There are two points I need to confirm if we >>> share the same problem. >>> >>> 1. Your job is using default restart strategy, which is per-second >>> restart. >>> 2. Your CPU resource on jobmanager might be small >>> >>> >>> >>> Here is some findings I want to share. >>> >>> ## Metaspace OOM >>> >>> Due to https://issues.apache.org/jira/browse/FLINK-15467 , when we have >>> some job restarts, there will be some threads from the sourceFunction >>> hanging, cause the class loader cannot close. New restarts would load new >>> classes, then expand the metaspace, and finally OOM happens. >>> >>> >>> >>> ## Leader retrieving >>> >>> Constant restarts may be heavy for jobmanager, if JM CPU resources are >>> not enough, the thread for leader retrieving may be stuck. >>> >>> >>> >>> Best Regards, >>> >>> Brian >>> >>> >>> >>> *From:* Xintong Song <tonysong...@gmail.com> >>> *Sent:* Tuesday, September 22, 2020 10:16 >>> *To:* Claude M; user >>> *Subject:* Re: metaspace out-of-memory & error while retrieving the >>> leader gateway >>> >>> >>> >>> ## Metaspace OOM >>> >>> As the error message already suggested, the metaspace OOM you >>> encountered is likely caused by a class loading leak. I think you are on >>> the right direction trying to look into the heap dump and find out where >>> the leak comes from. IIUC, after removing the ZK folder, you are now able >>> to run Flink with the heap dump options. >>> >>> >>> >>> The problem does not occur in previous versions because Flink starts to >>> set the metaspace limit since the 1.10 release. The class loading leak >>> might have already been there, but is never discovered. This could lead to >>> unpredictable stability and performance issues. That's why Flink updated >>> its memory model and explicitly set the metaspace limit in the 1.10 release. >>> >>> >>> >>> ## Leader retrieving >>> >>> The command looks good to me. If this problem happens only once, it >>> could be irrelevant to adding the options. If that does not block you from >>> getting the heap dump, we can look into it later. >>> >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> >>> >>> >>> On Mon, Sep 21, 2020 at 9:37 PM Claude M <claudemur...@gmail.com> wrote: >>> >>> Hi Xintong, >>> >>> >>> >>> Thanks for your reply. Here is the command output w/ the java.opts: >>> >>> >>> >>> /usr/local/openjdk-8/bin/java -Xms768m -Xmx768m -XX:+UseG1GC >>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/flink/log >>> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties >>> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml >>> -classpath >>> /opt/flink/lib/flink-metrics-datadog-statsd-2.11-0.1.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:/opt/flink/lib/flink-table-blink_2.11-1.10.2.jar:/opt/flink/lib/flink-table_2.11-1.10.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.15.jar:/opt/flink/lib/flink-dist_2.11-1.10.2.jar::/etc/hadoop/conf: >>> org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint >>> --configDir /opt/flink/conf --executionMode cluster >>> >>> >>> >>> To answer your questions: >>> >>> - Correct, in order for the pod to start up, I have to remove the >>> flink app folder from zookeeper. I only have to delete once after >>> applying >>> the java.opts arguments. It doesn't make sense though that I should have >>> to do this just from adding a parameter. >>> - I'm using the standalone deployment. >>> - I'm using job cluster mode. >>> >>> A higher priority issue I'm trying to solve is this metaspace out of >>> memory that is occuring in task managers. This was not happening before I >>> upgraded to Flink 1.10.2. Even after increasing the memory, I'm still >>> encountering the problem. That is when I added the java.opts argument to >>> see if I can get more information about the problem. That is when I ran >>> across the second issue w/ the job manager pod not starting up. >>> >>> >>> >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> On Sun, Sep 20, 2020 at 10:23 PM Xintong Song <tonysong...@gmail.com> >>> wrote: >>> >>> Hi Claude, >>> >>> >>> >>> IIUC, in your case the leader retrieving problem is triggered by adding >>> the `java.opts`? Then could you try to find and post the complete command >>> for launching the JVM process? You can try log into the pod and execute `ps >>> -ef | grep <PID>`. >>> >>> >>> >>> A few more questions: >>> >>> - What do you mean by "resolve this"? Does the jobmanager pod get stuck >>> there, and recover when you remove the folder from ZK? Do you have to do >>> the removal for everytime submitting the Kubernetes? >>> >>> The only way I can resolve this is to delete the folder from zookeeper >>> which I shouldn't have to do. >>> >>> - Which Flink's kubernetes deployment are you using? The standalone or >>> native Kubernetes? >>> >>> - Which cluster mode are you using? Job cluster, session cluster, or the >>> application mode? >>> >>> >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> >>> >>> >>> On Sat, Sep 19, 2020 at 1:22 AM Claude M <claudemur...@gmail.com> wrote: >>> >>> Hello, >>> >>> >>> >>> I upgraded from Flink 1.7.2 to 1.10.2. One of the jobs running on the >>> task managers is periodically crashing w/ the following error: >>> >>> >>> >>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error >>> has occurred. This can mean two things: either the job requires a larger >>> size of JVM metaspace to load classes or there is a class loading leak. In >>> the first case 'taskmanager.memory.jvm-metaspace.size' configuration option >>> should be increased. If the error persists (usually in cluster after >>> several job (re-)submissions) then there is probably a class loading leak >>> which has to be investigated and fixed. The task executor has to be >>> shutdown. >>> >>> >>> >>> I found this issue regarding it: >>> >>> https://issues.apache.org/jira/browse/FLINK-16406 >>> >>> >>> >>> I have tried increasing the taskmanager.memory.jvm-metaspace.size to >>> 256M & 512M and still was having the problem. >>> >>> >>> >>> I then added the following to the flink.conf to try to get more >>> information about the error: >>> >>> env.java.opts: -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError >>> -XX:HeapDumpPath=/opt/flink/log >>> >>> >>> >>> When I deployed the change which is in a Kubernetes cluster, the >>> jobmanager pod fails to start up and the following message shows >>> repeatedly: >>> >>> >>> >>> 2020-09-18 17:03:46,255 WARN >>> org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever - >>> Error while retrieving the leader gateway. Retrying to connect to >>> akka.tcp://flink@flink-jobmanager:50010/user/dispatcher. >>> >>> >>> >>> The only way I can resolve this is to delete the folder from zookeeper >>> which I shouldn't have to do. >>> >>> >>> >>> Any ideas on these issues? >>> >>> >>> >>> >>> >>> >>> >>>
