There are two things that you may want to try: - Configure 'state.backend.rocksdb.memory.managed' to true. That tells RocksDBStateBackend to try to limit its memory consumption within the managed memory size. - Increase JVM Overhead Memory Size (via 'taskmanager.memory.jvm-overhead.[min|max|fraction]'). That helps reserve more native memory in the Kubernetes pod.
Thank you~ Xintong Song On Fri, Jul 2, 2021 at 11:51 AM Sudharsan R <sud.r...@gmail.com> wrote: > Hi Xintong, > Thanks very much for the response. Let me check out the new UI on flink > 1.12. > > The reason I asked this question is because our flink cluster on k8s shows > a container_working_set_bytes(used by OOMkiller) to be > 3Gb. I assume that > the used(heap, non-heap) values on the UI are correct. If so (because heap > used + non-heap used + direct memory ~ 1Gb), either the rocksdb limit is > not being honored or kinesis producer is allocating memory which is not > showing up anywhere. Would you have any suggestions on how to understand > this? > > Thanks > Sudharsan > > On Thu, Jul 1, 2021 at 6:53 PM Xintong Song <tonysong...@gmail.com> wrote: > >> Hi Sudharsan, >> >> The non-heap max is decided by JVM automatically and is not controlled by >> Flink. Moreover, it doesn't mean Flink will use up to that size of non-heap >> memory. >> >> These metrics are fetched directly from JVM and do not correspond well >> with Flink's memory configurations, which very often lead to confusions. >> >> Since Flink-1.12, we have introduced a new web ui for the memory metrics, >> where the legacy metrics are preserved only for backward compatibility and >> are placed in an `Advanced` pane. I'd recommend ignoring them in 99% of the >> cases. >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Fri, Jul 2, 2021 at 2:34 AM Sudharsan R <sud.r...@gmail.com> wrote: >> >>> Hi, >>> >>> On my flink setup, I have taskmanager.memory.process.size set to 2536M. >>> I expect all the memory components shown on the UI to add up to this >>> number. However, I don't see this. >>> >>> >>> I have flink managed memory: 811Mb >>> >>> JVM heap max: 886Mb >>> >>> JVM non-heap max: 744Mb >>> >>> Direct memory: 204Mb >>> >>> >>> This adds up to 2645Mb. Am i adding up things i should not be? >>> >>> Also, how was JVM non-heap max derived to be 744Mb? >>> >>> >>> Thanks >>> >>> Sudharsan >>> >>> >>>
