Hi Pranjul, which deployment mode are you using?
- For session cluster, I'd say it's possible that memory grows with # of jobs. - For application mode, there is actual user-code executed, so if you're using some native libraries in your job driver, that may be another reason for the growing memory. If you want to debug the issue yourself, Yun already provided good pointers on how to approach that. Best, D. On Mon, Aug 16, 2021 at 8:38 AM Yun Tang <myas...@live.com> wrote: > Hi Pranjul, > > First of all, you adopted on-heap state backend: HashMapStateBackend, > which would not use native off-heap memory. Moreover, JobManager would not > initialize any keyed state backend instance. And if not enable high > availability, JobManagerCheckpointStorage would also not use direct memory > to write checkpoint stream out. Did you use some third-party native library? > > You could use native memory tracking [1] to see whether JVM used too much > overhead native memory. And use memory allocator analysis tool such as > JeMalloc [2] to see whether existed unexpected native memory usage. > > > [1] https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/ > [2] https://gist.github.com/thomasdarimont/79b3cef01e5786210309 > > Best > Yun Tang > ------------------------------ > *From:* Pranjul Ahuja <ahuja0...@gmail.com> > *Sent:* Monday, August 16, 2021 13:10 > *To:* user@flink.apache.org <user@flink.apache.org> > *Subject:* JobManager Resident memory Always Increasing > > Hi, > > We are running the JobManager container with 1024GB out of which 512MB is > allocated to the heap. We observe that the JobManager container's resident > memory is always increasing and it never comes down. Heap usage remains to > be constant and not rising abruptly. Can anyone help here where else this > memory is going? > > Statebackend Used - HashMapStateBackend > Checkpoint Storage - JobManagerCheckpointStorage > > >
