This is super cool. A couple of questions: - Right now you are using unpooled persistent memory. Does that make sense or does chunking make more sense? - What do you think is the right way to transition back and forth between persistent and ephemeral memory? What do you think will be the first pattern to be adopted. For example, do you think we should try to use it as a tiered storage for sort spilling (before hitting the disk), or should we use it for caching?
I think it will be much easier to think about this in the context of a primary or first use case. Do you have something in mind or should we brainstorm here? On Wed, Mar 23, 2016 at 7:16 PM, Gary <ga...@apache.org> wrote: > Hello, > > We have created a patch for Apache Arrow to leverage Apache incubator > Mnemonic as an alternative infra. for underlying memory resources > allocation, you can find it as below forked repo. > > https://github.com/NonVolatileComputing/arrow > > By this way, Apache Arrow could take some structural benefits from > Mnemonic project they are > > - Arrow is able to leverage larger capacity of high performance > hybrid storage devices. e.g. high-end SSD, NVMe > > - Mnemonic provide a potential opportunity for Arrow to > optimize/tuning its allocation algorithms as a native Arrow-oriented > allocation services > > - The non-volatile features of Mnemonic make it possible that Arrow > could make its columnar in-memory data shared between different > applications or across life-cycle of single application > > - Arrow could take advantages of coming Mnemonic features of memory > clustering/DOG (distributed object graph) and massive native computing > > - Mnemonic helps to reduce the pressure of main memory utilization > and its related system wide overheads. > > Our this patch is designed to minimize the changes for user to use > Arrow, please check out the test cases provided by this patch for your > reference. > > Note that, we need to put allocator services to a specified position > (indicated by pom.xml) for Mnemonic backed Arrow related test cases to > run because those services are required for external memory-like device > management. > > Please give your comments and review feedback for better > collaboration of Apache Arrow and Mnemonic, Thanks. > > Best Regards. > Gary. > > >
