Right after Job execution. Basically as soon as I deployed a 5th job. So at 4 jobs it was ok, at 5 jobs it would take like 1-2 minutes max and the node would just shut off. So far with MaxMetaSpace 256m it's been stable. My task nodes are 16GB and the memory config is done as follows... taskmanager.memory.flink.size: 12g taskmanager.memory.jvm-metaspace.size: 256m
100% of the jobs right now are ETL with checkpoints, NO state, Kafka -----> Json Transform ----> DB or Kafka ----> DB lookup (to small local cache)--------> Json Transform -----> Apache Ignite None of the jobs are related. On Mon, 24 Feb 2020 at 20:59, Xintong Song <tonysong...@gmail.com> wrote: > Hi John, > > The default metaspace size is intend for working with a major proportion > of jobs. We are aware that for some jobs that need to load lots of classes, > the default value might not be large enough. However, having a larger > default value means for other jobs that do not load many classes, the > overall memory requirements might be unnecessarily high. (Imagine you have > a task manager with the default total memory 1.5GB, but 512m of it is > reserved for metaspace.) > > Another possible problem is metaspace leak. When you say "eventually task > nodes started shutting down with OutOfMemory Metaspace", does this problem > happen shortly after the job execution starts, or does it happen after job > running for a while? Does the metaspace footprint keep growing or become > stable after the initial growth? If the metaspace keeps growing along with > time, it's usually an indicator of metaspace memory leak. > > Thank you~ > > Xintong Song > > > > On Tue, Feb 25, 2020 at 7:50 AM John Smith <java.dev....@gmail.com> wrote: > >> Hi, I just upgraded to 1.10 and I started deploying my jobs. Eventually >> task nodes started shutting down with OutOfMemory Metaspace. >> >> I look at the logs and the task managers are started with: >> -XX:MaxMetaspaceSize=100663296 >> >> So I configed: taskmanager.memory.jvm-metaspace.size: 256m >> >> It seems to be ok for now. What are your thoughts? And should I try 512m >> or is that too much? >> >
