Re: Question on releasing record batch

2021-06-17 Thread ZHOU Yuan
to hang onto memory due to > fragmentation issues, etc. Can you try measuring memory usage with > arrow::default_memory_pool()->bytes_allocated(); ? > > On Thu, Jun 17, 2021 at 3:48 PM ZHOU Yuan wrote: > > > > Hi Arrow developers, > > > > ran into a mem

Question on releasing record batch

2021-06-17 Thread ZHOU Yuan
Hi Arrow developers, ran into a memory footprint issue after releasing the record batch manually. The logic of my program is: 0. read many record batches 1. process on these batches 2. dump the intermediate results on disk 3. close the batches 4. logics for other operations I expect the memory fo

Java API for Arrow Compute

2019-11-25 Thread ZHOU Yuan
Hi Arrow developers, The compute APIs(neat!) seems now available in c++, python and rust, is there any plan/work on adding the Java API? Cheers, -yuan