Thanks for debugging and resolving the issue and driving the discussion
Yun!
For the given solutions, I prefer option 1 (supply another Dockerfile using
jemalloc as default memory allocator) because of the below reasons:
1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc),
or else glibc should have already adopted it as the default memory
allocator. And as indicated here [1], in some cases jemalloc will
consume as much as twice the memory than glibc
2. All existing Flink docker images use glibc, if we change the default
memory allocator to jemalloc and only supply one series of images, we will
leave those having better performance with glibc no other choices but
staying with old images. In another word, there's a risk of introducing new
problems while fixing an existing one if choosing option-2.
And there is a third option considering the efforts of maintaining more
images if the memory leak issue is not widely observed, that we could
document the steps of building Dockerfile with jemalloc as default
allocator so users could build it when needed, which leaves the burden to
our users so for me it's not the best option.
Best Regards,
Yu
[1] https://stackoverflow.com/a/33993215
On Tue, 13 Oct 2020 at 15:34, Yun Tang <myas...@live.com> wrote:
Hi all
Users report they meet serious memory leak when submitting jobs
continously in session mode within k8s (please refer to FLINK-18712[1] ),
and I also reproduce this to find this is caused by memory fragmentation
of
glibc [2][3] and provide solutions to fix this:
* Quick but not very clean solution to limit the memory pool of
glibc,
limit MALLOC_ARENA_MAX to 2
* More general solution by rebuilding the image to install
libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD
The reporter adopted the 2nd solution to fix this issue eventually. Thus,
I begin to think whether we should change our Dockerfile to adopt
jemalloc
as default memory allocator [4].
From my point of view, we have two choices:
1. Introduce another Dockerfile using jemalloc as default memory
allocator, which means Flink needs another two new image tags to build
docker with jemalloc while default docker still use glibc.
2. Set the default memory allocator as jemalloc in our existing
Dockerfiles, which means Flink offer docker image with jemalloc by
default.
I prefer the 2nd option as our company already use jemalloc as default
memory allocator for JDK at our production environment due to messages
from
os team warning of glibc's memory fragmentation.
Moreover, I found several open source projects adopting jemalloc as
default memory allocator within their images to resolve memory
fragmentation problem, e.g fluent [5], home-assistant [6].
What do you guys think of this issue?
[1] https://issues.apache.org/jira/browse/FLINK-18712
[2]
https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc
[3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321
[4] https://issues.apache.org/jira/browse/FLINK-19125
[5]
https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux
[6] https://github.com/home-assistant/core/pull/33237
Best
Yun Tang
