On 07/16/2018 11:35 AM, Claudio Freire wrote:
On Mon, Jul 16, 2018 at 11:34 AM Claudio Freire <klaussfre...@gmail.com> wrote:
On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan
<andrew.duns...@2ndquadrant.com> wrote:
On 07/13/2018 09:44 AM, Heikki Linnakangas wrote:
On 13/07/18 01:39, Andrew Dunstan wrote:
On 07/12/2018 06:34 PM, Alvaro Herrera wrote:
On 2018-Jul-12, Andrew Dunstan wrote:
I fully understand. I think this needs to go back to "Waiting on
Author".
Why? Heikki's patch applies fine and passes the regression tests.
Well, I understood Claudio was going to do some more work (see
upthread).
Claudio raised a good point, that doing small pallocs leads to
fragmentation, and in particular, it might mean that we can't give
back the memory to the OS. The default glibc malloc() implementation
has a threshold of 4 or 32 MB or something like that - allocations
larger than the threshold are mmap()'d, and can always be returned to
the OS. I think a simple solution to that is to allocate larger
chunks, something like 32-64 MB at a time, and carve out the
allocations for the nodes from those chunks. That's pretty
straightforward, because we don't need to worry about freeing the
nodes in retail. Keep track of the current half-filled chunk, and
allocate a new one when it fills up.
Google seems to suggest the default threshold is much lower, like 128K.
Still, making larger allocations seems sensible. Are you going to work
on that?
Below a few MB the threshold is dynamic, and if a block bigger than
128K but smaller than the higher threshold (32-64MB IIRC) is freed,
the dynamic threshold is set to the size of the freed block.
See M_MMAP_MAX and M_MMAP_THRESHOLD in the man page for mallopt[1]
So I'd suggest allocating blocks bigger than M_MMAP_MAX.
[1] http://man7.org/linux/man-pages/man3/mallopt.3.html
Sorry, substitute M_MMAP_MAX with DEFAULT_MMAP_THRESHOLD_MAX, the
former is something else.
Ah, ok. Thanks. ignore the email I just sent about that.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
