Update: I have done a memory profiling and the result seems to suggest memory leak. I have opened a issue to further discuss this: https://github.com/apache/arrow/issues/37630
On Fri, Sep 8, 2023 at 10:04 AM Li Jin <ice.xell...@gmail.com> wrote: > Update: > > I have done a memory profiling and the result seems to suggest memory > leak. I > have opened a issue to further discuss this: > https://github.com/apache/arrow/issues/37630 > > Attaching the memory profiling result here as well: > > On Wed, Sep 6, 2023 at 9:18 PM Gang Wu <ust...@gmail.com> wrote: > >> As suggested from other comments, I also highly recommend using a >> heap profiling tool to investigate what's going on there. >> >> BTW, 800 columns look suspicious to me. Could you try to test them >> without reading any batch? Not sure if the file metadata is the root >> cause. Or you may want to try another dataset with a smaller number >> of columns. >> >> On Thu, Sep 7, 2023 at 5:45 AM Li Jin <ice.xell...@gmail.com> wrote: >> >> > Correction: >> > >> > > I tried with both Antione's suggestions (swapping the default >> allocator >> > and calls ReleaseUnused but neither seem to affect the max rss. >> > >> > Calling ReleaseUnused does have some effect on the rss - the max rss >> goes >> > from ~6G -> 5G but still there seems to be something else. >> > >> > On Wed, Sep 6, 2023 at 4:35 PM Li Jin <ice.xell...@gmail.com> wrote: >> > >> > > Also attaching my experiment code just in case: >> > > https://gist.github.com/icexelloss/88195de046962e1d043c99d96e1b8b43 >> > > >> > > On Wed, Sep 6, 2023 at 4:29 PM Li Jin <ice.xell...@gmail.com> wrote: >> > > >> > >> Reporting back with some new findings. >> > >> >> > >> Re Felipe and Antione: >> > >> I tried with both Antione's suggestions (swapping the default >> allocator >> > >> and calls ReleaseUnused but neither seem to affect the max rss. In >> > >> addition, I manage to repro the issue by reading a list of n local >> > parquet >> > >> files that point to the same file, i.e., {"a.parquet", "a.parquet", >> ... >> > }. >> > >> I am also able to crash my process by reading and passing a large >> > enough n. >> > >> (I observed rss keep going up and eventually the process gets >> killed). >> > This >> > >> observation led me to think there might actually be some memory leak >> > issues. >> > >> >> > >> Re Xuwei: >> > >> Thanks for the tips. I am gonna try to memorize this profile next and >> > see >> > >> what I can find. >> > >> >> > >> I am gonna keep looking into this but again, any ideas / suggestions >> are >> > >> appreciated (and thanks for all the help so far!) >> > >> >> > >> Li >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> On Wed, Sep 6, 2023 at 1:59 PM Li Jin <ice.xell...@gmail.com> wrote: >> > >> >> > >>> Thanks all for the additional suggestions. Will try it but want to >> > >>> answer Antoine's question first: >> > >>> >> > >>> > Which leads to the question: what is your OS? >> > >>> >> > >>> I am testing this on Debian 5.4.228 x86_64 GNU/Linux >> > >>> >> > >>> On Wed, Sep 6, 2023 at 1:31 PM wish maple <maplewish...@gmail.com> >> > >>> wrote: >> > >>> >> > >>>> By the way, you can try to use a memory-profiler like [1] and [2] . >> > >>>> It would be help to find how the memory is used >> > >>>> >> > >>>> Best, >> > >>>> Xuwei Fu >> > >>>> >> > >>>> [1] >> > >>>> >> https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling >> > >>>> [2] https://google.github.io/tcmalloc/gperftools.html >> > >>>> >> > >>>> >> > >>>> Felipe Oliveira Carvalho <felipe...@gmail.com> 于2023年9月7日周四 >> 00:28写道： >> > >>>> >> > >>>> > > (a) stays pretty stable throughout the scan (stays < 1G), (b) >> > keeps >> > >>>> > increasing during the scan (looks linear to the number of files >> > >>>> scanned). >> > >>>> > >> > >>>> > I wouldn't take this to mean a memory leak but the memory >> allocator >> > >>>> not >> > >>>> > paging out virtual memory that has been allocated throughout the >> > scan. >> > >>>> > Could you run your workload under a memory profiler? >> > >>>> > >> > >>>> > (3) Scan the same dataset twice in the same process doesn't >> increase >> > >>>> the >> > >>>> > max rss. >> > >>>> > >> > >>>> > Another sign this isn't a leak, just the allocator reaching a >> level >> > of >> > >>>> > memory commitment that it doesn't feel like undoing. >> > >>>> > >> > >>>> > -- >> > >>>> > Felipe >> > >>>> > >> > >>>> > On Wed, Sep 6, 2023 at 12:56 PM Li Jin <ice.xell...@gmail.com> >> > wrote: >> > >>>> > >> > >>>> > > Hello, >> > >>>> > > >> > >>>> > > I have been testing "What is the max rss needed to scan through >> > >>>> ~100G of >> > >>>> > > data in a parquet stored in gcs using Arrow C++". >> > >>>> > > >> > >>>> > > The current answer is about ~6G of memory which seems a bit >> high >> > so >> > >>>> I >> > >>>> > > looked into it. What I observed during the process led me to >> think >> > >>>> that >> > >>>> > > there are some potential cache/memory issues in the >> > dataset/parquet >> > >>>> cpp >> > >>>> > > code. >> > >>>> > > >> > >>>> > > Main observation: >> > >>>> > > (1) As I am scanning through the dataset, I printed out (a) >> memory >> > >>>> > > allocated by the memory pool from ScanOptions (b) process rss. >> I >> > >>>> found >> > >>>> > that >> > >>>> > > while (a) stays pretty stable throughout the scan (stays < 1G), >> > (b) >> > >>>> keeps >> > >>>> > > increasing during the scan (looks linear to the number of files >> > >>>> scanned). >> > >>>> > > (2) I tested ScanNode in Arrow as well as an in-house library >> that >> > >>>> > > implements its own "S3Dataset" similar to Arrow dataset, both >> > >>>> showing >> > >>>> > > similar rss usage. (Which led me to think the issue is more >> likely >> > >>>> to be >> > >>>> > in >> > >>>> > > the parquet cpp code instead of dataset code). >> > >>>> > > (3) Scan the same dataset twice in the same process doesn't >> > >>>> increase the >> > >>>> > > max rss. >> > >>>> > > >> > >>>> > > I plan to look into the parquet cpp/dataset code but I wonder >> if >> > >>>> someone >> > >>>> > > has some clues what the issue might be or where to look at? >> > >>>> > > >> > >>>> > > Thanks, >> > >>>> > > Li >> > >>>> > > >> > >>>> > >> > >>>> >> > >>> >> > >> >
