Thank you for doing this Yijie, I encourage everyone else to have a look too -- there are some great ideas in both the document and the PR
Andrew On Fri, Nov 12, 2021 at 4:49 AM Yijie Shen <henry.yijies...@gmail.com> wrote: > Hi, > > I propose adding memory management capabilities to DataFusion, expanding > the scale of data it can handle, and increasing its applicable scenarios. > > Here's the design doc [1] available for comments, and an experimental PR > [2] illustrates the usage of the proposed API through an ExternalSorter > example. > > I want to raise it in this forum for a broader audience and more valuable > suggestions. > > [1] > > https://docs.google.com/document/d/1BT5HH-2sKq-Jxo51PNE6l9NNd_F-FyyYcyC3SKTnkIA/edit > [2] https://github.com/yjshen/arrow-datafusion/pull/3 > > > Best, > Yijie >
