Hello,
We run Flink 1.13.5 job in app mode in Kubernetes, 1 JM and 1 TM, we also have 
Kubernetes cron job which takes savepoint every 2 hour (14 */2 * * *), once in 
while (~1 per 2 days) TM is OOMKilled, suspiciously it happens on even hours ~4 
minutes after savepoint start (e.g. 12:18, 4:18) but I don't see failed save 
points, so I assume OOM happens right after savepoint taken. However OOMKilled 
doesn't happen on every save point, so maybe this is a random correlation.
I've reserved 2G for JVM overhead, but somehow it is not enough ? Any known 
issues with memory and savepoints? Any suggestions how to troubleshoot this?

 Final TaskExecutor Memory configuration:
   Total Process Memory:          10.000gb (10737418240 bytes)
     Total Flink Memory:          7.547gb (8103395328 bytes)
       Total JVM Heap Memory:     3.523gb (3783262149 bytes)
         Framework:               128.000mb (134217728 bytes)
         Task:                    3.398gb (3649044421 bytes)
       Total Off-heap Memory:     4.023gb (4320133179 bytes)
         Managed:                 3.019gb (3241358179 bytes)
         Total JVM Direct Memory: 1.005gb (1078775000 bytes)
           Framework:             128.000mb (134217728 bytes)
           Task:                  128.000mb (134217728 bytes)
           Network:               772.800mb (810339544 bytes)
     JVM Metaspace:               256.000mb (268435456 bytes)
     JVM Overhead:                2.203gb (2365587456 bytes)

Thanks,
Alexey

Reply via email to