On 19/06/2015 12:56, Alexandre Derumier wrote:
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
> to jemalloc memory allocator.
>
> We have already tcmalloc support,
> but it seem to not working well with a lot of iothreads/disks.
>
> The main problem is that tcmalloc use a shared thread cache of 16MB
> by default.
> With more threads, this cache is shared, and some bad garbage collections
> can occur if the cache is too low.
>
> It's possible to tcmalloc cache increase it with a env var:
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB
It's also possible to do
MallocExtension_SetNumericProperty("tcmalloc.max_total_thread_cache_bytes",
num_io_threads << 24);
What is the peak memory usage of jemalloc and tcmalloc?
Paolo
> With default 16MB, performances are really bad with more than 2 disks.
> Increasing to 256MB, it's helping but still have problem with 16
> disks/iothreads.
>
> Jemalloc don't have performance problem with default configuration.
>
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
> with rbd block backend (librbd is doing a lot of memory allocation),
> 1 iothread by disk
>
> glibc malloc
> ------------
>
> 1 disk 29052
> 2 disks 55878
> 4 disks 127899
> 8 disks 240566
> 15 disks 269976
>
> jemalloc
> --------
>
> 1 disk 41278
> 2 disks 75781
> 4 disks 195351
> 8 disks 294241
> 15 disks 298199
>
> tcmalloc 2.2.1 default 16M cache
> --------------------------------
>
> 1 disk 37911
> 2 disks 67698
> 4 disks 41076
> 8 disks 43312
> 15 disks 37569
>
> tcmalloc : 256M cache
> ---------------------------
>
> 1 disk 33914
> 2 disks 58839
> 4 disks 148205
> 8 disks 213298
> 15 disks 218383
>
> Signed-off-by: Alexandre Derumier <aderum...@odiso.com>
> ---
> configure | 29 +++++++++++++++++++++++++++++
> 1 file changed, 29 insertions(+)
>
> diff --git a/configure b/configure
> index 222694f..2fe1e05 100755
> --- a/configure
> +++ b/configure
> @@ -336,6 +336,7 @@ vhdx=""
> quorum=""
> numa=""
> tcmalloc="no"
> +jemalloc="no"
>
> # parse CC options first
> for opt do
> @@ -1147,6 +1148,10 @@ for opt do
> ;;
> --enable-tcmalloc) tcmalloc="yes"
> ;;
> + --disable-jemalloc) jemalloc="no"
> + ;;
> + --enable-jemalloc) jemalloc="yes"
> + ;;
> *)
> echo "ERROR: unknown option $opt"
> echo "Try '$0 --help' for more information"
> @@ -1420,6 +1425,8 @@ Advanced options (experts only):
> --enable-numa enable libnuma support
> --disable-tcmalloc disable tcmalloc support
> --enable-tcmalloc enable tcmalloc support
> + --disable-jemalloc disable jemalloc support
> + --enable-jemalloc enable jemalloc support
>
> NOTE: The object files are built at the place where configure is launched
> EOF
> @@ -3344,6 +3351,11 @@ EOF
> fi
> fi
>
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
> + echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
> + exit 1
> +fi
> +
> ##########################################
> # tcmalloc probe
>
> @@ -3361,6 +3373,22 @@ EOF
> fi
>
> ##########################################
> +# jemalloc probe
> +
> +if test "$jemalloc" = "yes" ; then
> + cat > $TMPC << EOF
> +#include <stdlib.h>
> +int main(void) { malloc(1); return 0; }
> +EOF
> +
> + if compile_prog "" "-ljemalloc" ; then
> + LIBS="-ljemalloc $LIBS"
> + else
> + feature_not_found "jemalloc" "install jemalloc devel"
> + fi
> +fi
> +
> +##########################################
> # signalfd probe
> signalfd="no"
> cat > $TMPC << EOF
> @@ -4499,6 +4527,7 @@ echo "snappy support $snappy"
> echo "bzip2 support $bzip2"
> echo "NUMA host support $numa"
> echo "tcmalloc support $tcmalloc"
> +echo "jemalloc support $jemalloc"
>
> if test "$sdl_too_old" = "yes"; then
> echo "-> Your SDL version is too old - please upgrade to have SDL support"
>
