Yes standalone cluster. 3 zoo, 3 job, 3 tasks. The task managers have taskslots at double core. So 2*4
I think metaspace of 2GB is ok. I'll try to get some jcmd stats. The jobs are fairly straight forward ETL they read from Kafka, do some json parsing, using vertx.io json parser and either Insert to apache ignite cache or jdbc db. On Sun., Dec. 26, 2021, 8:46 p.m. Xintong Song, <tonysong...@gmail.com> wrote: > Hi John, > > Sounds to me you have a Flink standalone cluster deployed directly on > physical hosts. If that is the case, use `t.m.flink.size` instead of > `t.m.process.size`. The latter does not limit the overall memory > consumption of the processes, and is only used for calculating how much > non-JVM memory the process should leave in a containerized setup, which > does no good in a non-containerized setup. > > When running into a Metaspace OOM, the standard solution is to increase > `t.m.jvm-metaspace.size`. If this is impractical due to the physical > limitations, you may also try to decrease `taskmanager.numberOfTaskSlots`. > If you have multiple jobs submitted to a shared Flink cluster, decreasing > the number of slots in a task manager should also reduce the amount of > classes loaded by the JVM, thus requiring less metaspace. > > Thank you~ > > Xintong Song > > > > On Mon, Dec 27, 2021 at 9:08 AM John Smith <java.dev....@gmail.com> wrote: > >> Ok I tried taskmanager.memory.process.size: 7168m >> >> It's worst, the task manager can barely start before it throws >> java.lang.OutOfMemoryError: Metaspace >> >> I will try... >> taskmanager.memory.flink.size: 5120m >> taskmanager.memory.jvm-metaspace.size: 2048m >> >> >> On Sun, 26 Dec 2021 at 19:46, John Smith <java.dev....@gmail.com> wrote: >> >>> Hi running Flink 1.10 >>> >>> I have >>> >>> taskmanager.memory.flink.size: 6144m >>> taskmanager.memory.jvm-metaspace.size: 1024m >>> taskmanager.numberOfTaskSlots: 8 >>> parallelism.default: 1 >>> >>> 1- The host has a physical ram of 8GB. I'm better off just to configure >>> "taskmanager.memory.process.size" as 7GB and let flink figure it out? >>> 2- Is there a way for me to calculate how much metspace my jobs require >>> or are using? >>> >>> 2021-12-24 04:53:32,511 ERROR >>> org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: >>> Thread 'flink-akka.actor.default-dispatcher-86' produced an uncaught >>> exception. Stopping the process... >>> java.lang.OutOfMemoryError: Metaspace >>> >>
