we also met glic memory leak .it appeared many 64 MB memory segment when using pmap command . i am +1 on setting jemalloc as default
发送自autohome ________________________________ 发件人: Yu Li <car...@gmail.com> 发送时间: 2020-10-30 00:15:12 收件人: dev <dev@flink.apache.org> 抄送: n...@ververica.com <n...@ververica.com> 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Thanks for sharing more thoughts guys. My experience with malloc libraries are also limited, and as mentioned in my previous email [1], my major concern comes from some online information [2] which indicates in some cases jemalloc will consume as much as twice the memory than glibc, which is a huge cost from my point of view. However, one may also argue that this post was written in 2015 and things might have been improved a lot ever since. Nevertheless, as Till and Yun pointed out, I could also see the benefits of setting jemalloc as default, so I don't have strong objections here. But this actually is a change of the default memory allocator of our docker image (previously glibc is the only choice) which definitely worth a big release note. Best Regards, Yu [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html [2] https://stackoverflow.com/a/33993215 On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <se...@apache.org<mailto:se...@apache.org>> wrote: > I think Till has a good point. The default should work well, because very > few users will ever change it. Even on OOMs. > So I am slightly in favor of going with jemalloc. > > @Yun Tang - Do you see a noticeable performance difference between the two? > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang > <myas...@live.com<mailto:myas...@live.com>> wrote: > > > Hi > > > > I think Till's view deserves wider discussion. > > I want to give my two cents when I debug with Nico on his reported > RocksDB > > OOM problem. > > Jemalloc has the mechanism to profile memory allocation [1] which is > > widely used to analysis memory leak. > > Once we set jemalloc as default memory allocator, the frequency of OOM > > behavior decreases obviously. > > > > Considering the OOM killed problem in k8s, change default memory > allocator > > as jemalloc could be something beneficial. > > > > [1] https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > Best > > Yun Tang > > > > ________________________________ > > From: Till Rohrmann <trohrm...@apache.org<mailto:trohrm...@apache.org>> > > Sent: Thursday, October 29, 2020 18:34 > > To: dev <dev@flink.apache.org<mailto:dev@flink.apache.org>> > > Cc: Yun Tang <myas...@live.com<mailto:myas...@live.com>> > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > > for debian based Flink docker image > > > > Hi Yu, > > > > I see your point why you are in favour of continuing using glibc. > > > > I honestly don't have a lot of experience with malloc libraries so I can > > only argue from a general perspective: We know that glibc has some > problems > > wrt to memory fragmentation which can cause processes to exceed its > memory > > limit. jemalloc seems to address this problem by avoiding memory > > fragmentation. However, it might be the case that jemalloc introduces > other > > problems. In order to figure this out we would need to collect more data > > points/experience. So in the end we have to weigh glibc memory > > fragmentation against unknown problems in jemalloc. > > > > Given that choosing the malloc library will be quite an expert setting, I > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > would be slightly in favour of choosing jemalloc as the default because > it > > would allow us to gather feedback from users and solve the existing > > problem. In the best case, user's won't run into new problems. If they > do, > > then we will have to re-evaluate this decision and might have to switch > the > > malloc library back. W/o setting jemalloc as the default, I fear that we > > will never gather enough feedback to make jemalloc confidently the > default. > > > > What one could argue, though, is that making jemalloc the default so > > shortly before the feature freeze limits the timespan we can observe how > it > > behaves before the actual release. Doing such a change at the beginning > of > > a release cycle, would allow us to gain more confidence. > > > > In any way, I believe that we should add a big release note and document > > what users should do in case they see memory issues (OOM kills, slower > > performance, etc.). > > > > Cheers, > > Till > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li > > <car...@gmail.com<mailto:car...@gmail.com>> wrote: > > > > > True, thanks for the reminder Till! > > > > > > I suggest using glibc malloc as default. On one hand this follows our > old > > > behavior (only with glibc malloc support in the image), on the other > > hand I > > > believe glibc isn't using jemalloc as its default memory allocator for > > some > > > reason. > > > > > > Please let me know your thoughts. Thanks. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann > > > <trohrm...@apache.org<mailto:trohrm...@apache.org>> > > wrote: > > > > > > > The only question left would be what will be the default value? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li > > > > <car...@gmail.com<mailto:car...@gmail.com>> wrote: > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > It seems we have reached consensus and there are already enough +1s > > to > > > > move > > > > > forward, and suggest @Yun to conclude the discussion directly if > > there > > > > are > > > > > no objections. > > > > > > > > > > Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann > > > > > <trohrm...@apache.org<mailto:trohrm...@apache.org>> > > > > wrote: > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > ches...@apache.org<mailto:<br></a>> > > ches...@apache.org>> > > > > > > wrote: > > > > > > > > > > > > > If it is possible to support both allocators in a single image > > then > > > > we > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > Thanks for Yang's suggestion. I think this could be a better > > > > choice. > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > when > > > > user > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > By doing so, we could avoid to create another docker image > tags > > > and > > > > > > also > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > Best > > > > > > > > Yun Tang > > > > > > > > ________________________________ > > > > > > > > From: Yang Wang > > > > > > > > <danrtsey...@gmail.com<mailto:danrtsey...@gmail.com>> > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > To: dev <dev@flink.apache.org<mailto:dev@flink.apache.org>> > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > memory > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > container > > > > > > > environment > > > > > > > > in production. I just > > > > > > > > have quick question. Could we have both memory allocator(e.g. > > > > glibc, > > > > > > > > jemalloc) in the Flink > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > Best, > > > > > > > > Yang > > > > > > > > > > > > > > > > Yu Li <car...@gmail.com<mailto:car...@gmail.com>> > > > > > > > > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > > > discussion > > > > > > > >> Yun! > > > > > > > >> > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > Dockerfile > > > > > > > using > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > reasons: > > > > > > > >> > > > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > > > (glibc > > > > > > > malloc), > > > > > > > >> or else glibc should have already adopted it as the default > > > memory > > > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > > > will > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > >> > > > > > > > >> 2. All existing Flink docker images use glibc, if we change > > the > > > > > > default > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > images, > > > > > we > > > > > > > will > > > > > > > >> leave those having better performance with glibc no other > > > choices > > > > > but > > > > > > > >> staying with old images. In another word, there's a risk of > > > > > > introducing > > > > > > > new > > > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > > > >> > > > > > > > >> And there is a third option considering the efforts of > > > maintaining > > > > > > more > > > > > > > >> images if the memory leak issue is not widely observed, that > > we > > > > > could > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > default > > > > > > > >> allocator so users could build it when needed, which leaves > > the > > > > > burden > > > > > > > to > > > > > > > >> our users so for me it's not the best option. > > > > > > > >> > > > > > > > >> Best Regards, > > > > > > > >> Yu > > > > > > > >> > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > >> > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang > > > > > > > >> <myas...@live.com<mailto:myas...@live.com>> > > > wrote: > > > > > > > >> > > > > > > > >>> Hi all > > > > > > > >>> > > > > > > > >>> Users report they meet serious memory leak when submitting > > jobs > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > FLINK-18712[1] > > > > > > > ), > > > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > > > fragmentation > > > > > > > >> of > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > >>> > > > > > > > >>> * Quick but not very clean solution to limit the > memory > > > pool > > > > > of > > > > > > > >> glibc, > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > >>> > > > > > > > >>> * More general solution by rebuilding the image to > > install > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > > > >>> > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > eventually. > > > > > > > Thus, > > > > > > > >>> I begin to think whether we should change our Dockerfile to > > > adopt > > > > > > > >> jemalloc > > > > > > > >>> as default memory allocator [4]. > > > > > > > >>> From my point of view, we have two choices: > > > > > > > >>> > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > default > > > > > memory > > > > > > > >>> allocator, which means Flink needs another two new image > tags > > > to > > > > > > build > > > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > > > existing > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > jemalloc > > > > by > > > > > > > >> default. > > > > > > > >>> I prefer the 2nd option as our company already use jemalloc > > as > > > > > > default > > > > > > > >>> memory allocator for JDK at our production environment due > to > > > > > > messages > > > > > > > >> from > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > >>> Moreover, I found several open source projects adopting > > > jemalloc > > > > as > > > > > > > >>> default memory allocator within their images to resolve > > memory > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > > > >>> > > > > > > > >>> What do you guys think of this issue? > > > > > > > >>> > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > >>> [2] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > >>> [5] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Best > > > > > > > >>> Yun Tang > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
