[
https://issues.apache.org/jira/browse/ARROW-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rok Mihevc updated ARROW-5429:
------------------------------
External issue URL: https://github.com/apache/arrow/issues/21882
> [Java] Provide alternative buffer allocation policy
> ---------------------------------------------------
>
> Key: ARROW-5429
> URL: https://issues.apache.org/jira/browse/ARROW-5429
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Java
> Reporter: Liya Fan
> Assignee: Liya Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The current buffer allocation policy works like this:
> * If the requested buffer size is greater than or equal to the chunk size,
> the buffer size will be as is.
> * If the requested size is within the chunk size, the buffer size will be
> rounded to the next power of 2.
> This policy can lead to waste of memory in some cases. For example, if we
> request a buffer of size 10MB, Arrow will round the buffer size to 16 MB. If
> we only need 10 MB, this will lead to a waste of (16 - 10) / 10 = 60% of
> memory.
> So in this proposal, we provide another policy: the rounded buffer size must
> be a multiple of some memory unit, like (32 KB). This policy has two benefits:
> # The wasted memory cannot exceed one memory unit (32 KB), which is much
> smaller than the power-of-two policy.
> # This is the memory allocation policy adopted by some computation engines
> (e.g. Apache Flink).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
