Hi > Do you see a noticeable performance difference between the two? @ Stephan Ewen , as we already use jemalloc as default memory allocator in production, we do not have much experience to compare performace between glibc and jemalloc. And I did not take a look at the performance difference when I debug docker OOM. I'll have a try to run benchmark on docker with different allocators when I have time these days.
@wang gang, yes, that is what I also observed when I pmap the memory when using glibc, many memory segments with 64MB size. @ Billy Xie, what kind of test case did you use? In my point of view, compared to who would use more memory in some cases, we should care more about who would behave under the max limit for most cases. Best Yun Tang ________________________________ From: Xie Billy <billy.xiezhi...@gmail.com> Sent: Friday, October 30, 2020 8:46 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image After long time running test application(>96 hours) , glibc used less memory than jemalloc , almost half of jemalloc. Can refer this test case. Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 12:38 AM 王刚 <wanggang11...@autohome.com.cn> wrote: > we also met glic memory leak .it appeared many 64 MB memory segment when > using pmap command . i am +1 on setting jemalloc as default > > 发送自autohome > ________________________________ > 发件人: Yu Li <car...@gmail.com> > 发送时间: 2020-10-30 00:15:12 > 收件人: dev <dev@flink.apache.org> > 抄送: n...@ververica.com <n...@ververica.com> > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > debian based Flink docker image > > Thanks for sharing more thoughts guys. > > My experience with malloc libraries are also limited, and as mentioned in > my previous email [1], my major concern comes from some online information > [2] which indicates in some cases jemalloc will consume as much as twice > the memory than glibc, which is a huge cost from my point of view. However, > one may also argue that this post was written in 2015 and things might have > been improved a lot ever since. > > Nevertheless, as Till and Yun pointed out, I could also see the benefits of > setting jemalloc as default, so I don't have strong objections here. But > this actually is a change of the default memory allocator of our docker > image (previously glibc is the only choice) which definitely worth a big > release note. > > Best Regards, > Yu > > [1] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > [2] https://stackoverflow.com/a/33993215 > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <se...@apache.org<mailto: > se...@apache.org>> wrote: > > > I think Till has a good point. The default should work well, because very > > few users will ever change it. Even on OOMs. > > So I am slightly in favor of going with jemalloc. > > > > @Yun Tang - Do you see a noticeable performance difference between the > two? > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <myas...@live.com<mailto: > myas...@live.com>> wrote: > > > > > Hi > > > > > > I think Till's view deserves wider discussion. > > > I want to give my two cents when I debug with Nico on his reported > > RocksDB > > > OOM problem. > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > widely used to analysis memory leak. > > > Once we set jemalloc as default memory allocator, the frequency of OOM > > > behavior decreases obviously. > > > > > > Considering the OOM killed problem in k8s, change default memory > > allocator > > > as jemalloc could be something beneficial. > > > > > > [1] > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > Best > > > Yun Tang > > > > > > ________________________________ > > > From: Till Rohrmann <trohrm...@apache.org<mailto:trohrm...@apache.org > >> > > > Sent: Thursday, October 29, 2020 18:34 > > > To: dev <dev@flink.apache.org<mailto:dev@flink.apache.org>> > > > Cc: Yun Tang <myas...@live.com<mailto:myas...@live.com>> > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > allocator > > > for debian based Flink docker image > > > > > > Hi Yu, > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > I honestly don't have a lot of experience with malloc libraries so I > can > > > only argue from a general perspective: We know that glibc has some > > problems > > > wrt to memory fragmentation which can cause processes to exceed its > > memory > > > limit. jemalloc seems to address this problem by avoiding memory > > > fragmentation. However, it might be the case that jemalloc introduces > > other > > > problems. In order to figure this out we would need to collect more > data > > > points/experience. So in the end we have to weigh glibc memory > > > fragmentation against unknown problems in jemalloc. > > > > > > Given that choosing the malloc library will be quite an expert > setting, I > > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > > would be slightly in favour of choosing jemalloc as the default because > > it > > > would allow us to gather feedback from users and solve the existing > > > problem. In the best case, user's won't run into new problems. If they > > do, > > > then we will have to re-evaluate this decision and might have to switch > > the > > > malloc library back. W/o setting jemalloc as the default, I fear that > we > > > will never gather enough feedback to make jemalloc confidently the > > default. > > > > > > What one could argue, though, is that making jemalloc the default so > > > shortly before the feature freeze limits the timespan we can observe > how > > it > > > behaves before the actual release. Doing such a change at the beginning > > of > > > a release cycle, would allow us to gain more confidence. > > > > > > In any way, I believe that we should add a big release note and > document > > > what users should do in case they see memory issues (OOM kills, slower > > > performance, etc.). > > > > > > Cheers, > > > Till > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <car...@gmail.com<mailto: > car...@gmail.com>> wrote: > > > > > > > True, thanks for the reminder Till! > > > > > > > > I suggest using glibc malloc as default. On one hand this follows our > > old > > > > behavior (only with glibc malloc support in the image), on the other > > > hand I > > > > believe glibc isn't using jemalloc as its default memory allocator > for > > > some > > > > reason. > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <trohrm...@apache.org > <mailto:trohrm...@apache.org>> > > > wrote: > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <car...@gmail.com<mailto: > car...@gmail.com>> wrote: > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > It seems we have reached consensus and there are already enough > +1s > > > to > > > > > move > > > > > > forward, and suggest @Yun to conclude the discussion directly if > > > there > > > > > are > > > > > > no objections. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Best Regards, > > > > > > Yu > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > trohrm...@apache.org<mailto:trohrm...@apache.org>> > > > > > wrote: > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > ches...@apache.org<mailto:<br></a>> > > ches...@apache.org>> > > > > > > > wrote: > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > image > > > then > > > > > we > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > better > > > > > choice. > > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > > when > > > > > user > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > By doing so, we could avoid to create another docker image > > tags > > > > and > > > > > > > also > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > Best > > > > > > > > > Yun Tang > > > > > > > > > ________________________________ > > > > > > > > > From: Yang Wang <danrtsey...@gmail.com<mailto: > danrtsey...@gmail.com>> > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > To: dev <dev@flink.apache.org<mailto:dev@flink.apache.org > >> > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > memory > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > container > > > > > > > > environment > > > > > > > > > in production. I just > > > > > > > > > have quick question. Could we have both memory > allocator(e.g. > > > > > glibc, > > > > > > > > > jemalloc) in the Flink > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > Yu Li <car...@gmail.com<mailto:car...@gmail.com>> > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > the > > > > > > > discussion > > > > > > > > >> Yun! > > > > > > > > >> > > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > > Dockerfile > > > > > > > > using > > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > > reasons: > > > > > > > > >> > > > > > > > > >> 1. It's hard to say jemalloc is always better than > ptmalloc > > > > (glibc > > > > > > > > malloc), > > > > > > > > >> or else glibc should have already adopted it as the > default > > > > memory > > > > > > > > >> allocator. And as indicated here [1], in some cases > jemalloc > > > > will > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > >> > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > change > > > the > > > > > > > default > > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > > images, > > > > > > we > > > > > > > > will > > > > > > > > >> leave those having better performance with glibc no other > > > > choices > > > > > > but > > > > > > > > >> staying with old images. In another word, there's a risk > of > > > > > > > introducing > > > > > > > > new > > > > > > > > >> problems while fixing an existing one if choosing > option-2. > > > > > > > > >> > > > > > > > > >> And there is a third option considering the efforts of > > > > maintaining > > > > > > > more > > > > > > > > >> images if the memory leak issue is not widely observed, > that > > > we > > > > > > could > > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > > default > > > > > > > > >> allocator so users could build it when needed, which > leaves > > > the > > > > > > burden > > > > > > > > to > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > >> > > > > > > > > >> Best Regards, > > > > > > > > >> Yu > > > > > > > > >> > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > >> > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <myas...@live.com > <mailto:myas...@live.com>> > > > > wrote: > > > > > > > > >> > > > > > > > > >>> Hi all > > > > > > > > >>> > > > > > > > > >>> Users report they meet serious memory leak when > submitting > > > jobs > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > FLINK-18712[1] > > > > > > > > ), > > > > > > > > >>> and I also reproduce this to find this is caused by > memory > > > > > > > > fragmentation > > > > > > > > >> of > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > >>> > > > > > > > > >>> * Quick but not very clean solution to limit the > > memory > > > > pool > > > > > > of > > > > > > > > >> glibc, > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > >>> > > > > > > > > >>> * More general solution by rebuilding the image to > > > install > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > LD_PRELOAD > > > > > > > > >>> > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > eventually. > > > > > > > > Thus, > > > > > > > > >>> I begin to think whether we should change our Dockerfile > to > > > > adopt > > > > > > > > >> jemalloc > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > >>> > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > default > > > > > > memory > > > > > > > > >>> allocator, which means Flink needs another two new image > > tags > > > > to > > > > > > > build > > > > > > > > >>> docker with jemalloc while default docker still use > glibc. > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > our > > > > > existing > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > jemalloc > > > > > by > > > > > > > > >> default. > > > > > > > > >>> I prefer the 2nd option as our company already use > jemalloc > > > as > > > > > > > default > > > > > > > > >>> memory allocator for JDK at our production environment > due > > to > > > > > > > messages > > > > > > > > >> from > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > jemalloc > > > > > as > > > > > > > > >>> default memory allocator within their images to resolve > > > memory > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > [6]. > > > > > > > > >>> > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > >>> > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > >>> [2] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > >>> [3] > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > >>> [5] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> Best > > > > > > > > >>> Yun Tang > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
