I think you can try to increase the JVM metaspace option for TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace Best, Yangze Guo Best, Yangze Guo On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <495635...@qq.com> wrote: > > batch job： > read data from s3 by sql，then by some operators and write data to clickhouse > and kafka. > after some times, task-manager quit with OutOfMemoryError: Metaspace. > > env： > flink version：1.12.2 > task-manager slot count: 5 > deployment： standalone kubernetes session 模式 > dependencies： > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-connector-kafka_2.11</artifactId> > > <version>${flink.version}</version> > > </dependency> > > <dependency> > > <groupId>com.google.code.gson</groupId> > > <artifactId>gson</artifactId> > > <version>2.8.5</version> > > </dependency> > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-connector-jdbc_2.11</artifactId> > > <version>${flink.version}</version> > > </dependency> > > <dependency> > > <groupId>ru.yandex.clickhouse</groupId> > > <artifactId>clickhouse-jdbc</artifactId> > > <version>0.3.0</version> > > </dependency> > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-parquet_2.11</artifactId> > > <version>${flink.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.flink</groupId> > > <artifactId>flink-json</artifactId> > > <version>${flink.version}</version> > > </dependency> > > > heap dump1: > > Leak Suspects > > System Overview > > Leaks > > Overview > > > Problem Suspect 1 > > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by > "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) > bytes. > > Biggest instances: > > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) > bytes. > > > > Keywords > org.apache.flink.util.ChildFirstClassLoader > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0 > Details » > > Problem Suspect 2 > > 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", > loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 > (10.70%) bytes. > > Keywords > org.apache.flink.core.memory.HybridMemorySegment > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0 > > Details » > > > > heap dump2: > > Leak Suspects > > System Overview > > Leaks > > Overview > > Problem Suspect 1 > > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by > "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) > bytes. > > Biggest instances: > > org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) > bytes. > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) > bytes. > > > > Keywords > org.apache.flink.util.ChildFirstClassLoader > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0 > Details » > > Problem Suspect 2 > > 4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", > loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 > (13.71%) bytes. > > Biggest instances: > > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - > 4,364,536 (5.14%) bytes. > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - > 3,643,576 (4.29%) bytes. > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - > 3,635,952 (4.28%) bytes. > > > > Keywords > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0 > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask > Details » > >
