lhotari opened a new pull request, #24692: URL: https://github.com/apache/pulsar/pull/24692
### Motivation One of the drawbacks of the Alpine based image is that it's based on the "musl" libc library which causes incompatibilities for binaries that are compiled for glibc library. In Pulsar, we have already resolved this for binaries distributed with Pulsar and Pulsar IO connectors. However, there's another disadvantage. The memory allocation library (malloc) in Alpine doesn't contain any tuneables and in Alpine, there's no documentation how to tune malloc operations. It's possible to replace the default malloc library with other allocation libraries, but this could cause other instability. For benchmarking purposes, it would be useful to be able to build a glibc based Docker image for Pulsar and specify [glibc memory allocation tuneables](https://sourceware.org/glibc/manual/latest/html_node/Memory-Allocation-Tunables.html). Various glibc tuneables can be viewed with this command: ``` docker run --rm -it cgr.dev/chainguard/wolfi-base sh -c "/lib64/ld-linux-*.so.1 --list-tunables" ``` The reference: https://sourceware.org/glibc/manual/latest/html_node/Memory-Allocation-Tunables.html For example, to enable Transparent Huge Pages (THP) for malloc allocations: ``` export GLIBC_TUNABLES=glibc.malloc.hugetlb=1 ``` Great resources about Transparent Huge Pages in Linux: * https://www.oxla.com/blog/using-huge-pages-in-linux-applications * https://www.oxla.com/blog/using-huge-pages-in-linux-applications-part-2-transparent-hugepage For performance testing, it might be useful to also set `glibc.malloc.mmap_threshold` to the bytes value of 2MB: ``` export GLIBC_TUNABLES=glibc.malloc.hugetlb=1:glibc.malloc.mmap_threshold=2097152 ``` For k8s or Docker configurations `GLIBC_TUNABLES` would be passed with the value `glibc.malloc.hugetlb=1:glibc.malloc.mmap_threshold=2097152`. The reason why this matters is that most of BookKeeper's and Pulsar's memory is allocated by Netty. Under the covers the allocations are made using `os::malloc` which maps to the glibc `malloc`. To get the performance benefits of Transparent Huge Pages, it's necessary that the allocations to the operating system are made using mmap with madvise for `MADV_HUGEPAGE`. The Java VM will handle this for the Java heap memory with `-XX:+TransparentHugePages`, but that's not sufficient, since it doesn't cover the allocations made by Netty and RocksDB. When the malloc library has support for THP, there are better ways to control the behavior. When using glibc, it will mark the large allocations with `MADV_HUGEPAGE` when `glibc.malloc.hugetlb=1` is used and closes that gap for Pulsar. THP is also supported in many other allocators and for example Microsoft's mimalloc (https://github.com/microsoft/mimalloc) could be another option. However, I faced some stability issues with it on Alpine. It's possibly an issue when `LD_PRELOAD` contains both `gcompat` and `mimalloc` libraries. In the Alpine images, `gcompat` is required for supporting Netty native libraries which have been built for glibc. This compatibility issue is also solved by using the Wolfi image. ### Modifications - add additional Dockerfiles `Dockerfile.wolfi` for `pulsar` and `pulsar-all` images. - make changes to the maven build that Wolfi based images will be built when `-Pdocker-wolfi` is passed to the command line ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
