You don’t (shouldn’t) need to rebuild the binary to use jemalloc. It should be possible to do something like
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ceph-osd … The last time we tried it segfaulted after a few minutes, so YMMV and be careful. Jan > On 23 Jul 2015, at 18:18, Luis Periquito <periqu...@gmail.com> wrote: > > Hi Greg, > > I've been looking at the tcmalloc issues, but did seem to affect osd's, and I > do notice it in heavy read workloads (even after the patch and increasing > TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728). This is affecting the mon > process though. > > looking at perf top I'm getting most of the CPU usage in mutex lock/unlock > 5.02% libpthread-2.19.so <http://libpthread-2.19.so/> [.] > pthread_mutex_unlock > 3.82% libsoftokn3.so [.] 0x000000000001e7cb > 3.46% libpthread-2.19.so <http://libpthread-2.19.so/> [.] > pthread_mutex_lock > > I could try to use jemalloc, are you aware of any built binaries? Can I mix a > cluster with different malloc binaries? > > > On Thu, Jul 23, 2015 at 10:50 AM, Gregory Farnum <g...@gregs42.com > <mailto:g...@gregs42.com>> wrote: > On Thu, Jul 23, 2015 at 8:39 AM, Luis Periquito <periqu...@gmail.com > <mailto:periqu...@gmail.com>> wrote: > > The ceph-mon is already taking a lot of memory, and I ran a heap stats > > ------------------------------------------------ > > MALLOC: 32391696 ( 30.9 MiB) Bytes in use by application > > MALLOC: + 27597135872 (26318.7 MiB) Bytes in page heap freelist > > MALLOC: + 16598552 ( 15.8 MiB) Bytes in central cache freelist > > MALLOC: + 14693536 ( 14.0 MiB) Bytes in transfer cache freelist > > MALLOC: + 17441592 ( 16.6 MiB) Bytes in thread cache freelists > > MALLOC: + 116387992 ( 111.0 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 27794649240 (26507.0 MiB) Actual memory used (physical + swap) > > MALLOC: + 26116096 ( 24.9 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 27820765336 (26531.9 MiB) Virtual address space used > > MALLOC: > > MALLOC: 5683 Spans in use > > MALLOC: 21 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > > > after that I ran the heap release and it went back to normal. > > ------------------------------------------------ > > MALLOC: 22919616 ( 21.9 MiB) Bytes in use by application > > MALLOC: + 4792320 ( 4.6 MiB) Bytes in page heap freelist > > MALLOC: + 18743448 ( 17.9 MiB) Bytes in central cache freelist > > MALLOC: + 20645776 ( 19.7 MiB) Bytes in transfer cache freelist > > MALLOC: + 18456088 ( 17.6 MiB) Bytes in thread cache freelists > > MALLOC: + 116387992 ( 111.0 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 201945240 ( 192.6 MiB) Actual memory used (physical + swap) > > MALLOC: + 27618820096 <tel:%2B%20%2027618820096> (26339.4 MiB) Bytes > > released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 27820765336 (26531.9 MiB) Virtual address space used > > MALLOC: > > MALLOC: 5639 Spans in use > > MALLOC: 29 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > > > So it just seems the monitor is not returning unused memory into the OS or > > reusing already allocated memory it deems as free... > > Yep. This is a bug (best we can tell) in some versions of tcmalloc > combined with certain distribution stacks, although I don't think > we've seen it reported on Trusty (nor on a tcmalloc distribution that > new) before. Alternatively some folks are seeing tcmalloc use up lots > of CPU in other scenarios involving memory return and it may manifest > like this, but I'm not sure. You could look through the mailing list > for information on it. > -Greg > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
