Uwe L. Korn created ARROW-1123:
----------------------------------
Summary: C++: Make jemalloc the default allocator
Key: ARROW-1123
URL: https://issues.apache.org/jira/browse/ARROW-1123
Project: Apache Arrow
Issue Type: New Feature
Components: C++
Reporter: Uwe L. Korn
While one of the advertised point of Arrow Java is that it comes with an
efficient allocator that is based on jemalloc, we only provide jemalloc as an
alternative allocator in C++. I would propose to switch to it as the default
allocator on all platforms jemalloc is available (i.e. MacOS and Linux) as it
provides the following benefits:
* in-place resizing while ensuring 64byte alignment, for allocations >4kb this
won't touch the data but rather only changes some pointers in kernel internal
structs pointing to the respecitive memory pages.
* better allocation algorithm, see
https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919/
that should lead to less fragmentation and support fast multithreaded
allocations
Especially for the builders, simply changing the allocator shows speedups
between 2x and 10x. In the optimal case, this puts builders nearly on the same
level as if one would have pre-allocated the final size.
This change should also include a blog post that outlines the benefits.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
