Hi Weston, thank you for the inputs!
I was watching the memory usage using "top -p `pidof test`", the size of
the residence memory is not reduced.
With the new counter I saw the memory is freed immediately on arrow side.
So this is related to my allocator.
I actually disbaled jemalloc/mimalloc du
The only owner of input_batch that I can see here is the shared_ptr
that you are resetting so I would expect the memory to be freed.
How are you measuring memory usage? The dynamic allocators (mimalloc
/ jemalloc) don't always release memory as soon as they possibly can.
Even malloc will sometime
Hi Arrow developers,
ran into a memory footprint issue after releasing the record batch
manually. The logic of my program is:
0. read many record batches
1. process on these batches
2. dump the intermediate results on disk
3. close the batches
4. logics for other operations
I expect the memory fo
