Hi John, I recently proposed on the mailing list an experimental extension of the Arrow IPC protocol that would make it easier to leverage disaggregated shared memory along with non-cpu memory via utilities such as UCX and libfabric [1]. I'll be putting together a more formal description of it that will be being added to the Arrow website, but would still love any feedback you might have for it and how it could work with Famfs. It would be interesting if we could put together a POC leveraging CXL, Famfs, and the proposed protocol. There were a few suggestions and responses to the document that included use-cases from interested individuals such as being able to stick an Arrow record batch into shared memory and update it in-place.
I think the best way to get the word out and bring interest to this would be to create a really great POC showing a particular use-case being significantly improved by utilizing all of these pieces. I'd love to see if we could work together and construct such a POC and get some benchmarking numbers for comparisons. A great place to start would be comparing the disaggregated shared memory performance against various network transfer / RPC protocols for some use cases processing large amounts of data in a distributed system. What do you think? --Matt [1]: https://docs.google.com/document/d/1zHbnyK1r6KHpMOtEdIg1EZKNzHx-MVgUMOzB87GuXyk/edit#heading=h.38515dnp2bdb On Tue, Apr 9, 2024 at 10:14 PM John Groves <j...@groves.net> wrote: > This is a request for comments from the Arrow developer community. > > I’m reaching out to start making the Arrow community aware of work that my > team at Micron has recently open-sourced. Because of the Compute Express > Link (CXL) standard, sharable disaggregated memory is coming – this is > memory shared by multiple nodes in a cluster. Arrow and the other > zero-copy formats are a great fit for shared memory if a natural enough > access method emerges. > > That’s where famfs comes in. (Famfs stands for Fabric-Attached Memory File > System.) Famfs supports formatting shared memory as a file system that can > be simultaneously mounted from multiple hosts. > > Putting zero-copy data frames in famfs files allows jobs across a cluster > to memory map data frames from a single copy in shared memory. This has the > potential to deduplicate memory while reducing or avoiding sharding and > shuffling overheads. > > Famfs files can be memory-mapped and used without awareness that the files > are “special” (though creating famfs files does require special steps). > Memory mapping a famfs file provides direct access to the memory – with no > copying through the page cache. > > Famfs was published in February as a Linux kernel patch > < > https://lore.kernel.org/linux-fsdevel/ze5edu3jblefw...@dread.disaster.area/T/#m27639915e97443186b3ade9d1e94423bc58e6e22 > > > and a user space CLI and library; all are available on github > <https://github.com/cxl-micron-reskit/famfs/blob/master/README.md>. The > kernel patch set has been received seriously; if legitimate use cases are > demonstrated, we expect it will make its way into mainline Linux – and we > intend to step up and maintain it. > > Famfs is already usable with shared disaggregated memory (though this > memory is not commercially available yet). Conventional memory can be > shared among virtual machines today, to build (admittedly scaled down) > POCs. > > I am looking for the following feedback: > > - Any questions are welcome, on or off-list. > - Please tell us what sorts of work flows you might try with famfs > shared memory if you had it – we are looking for ways to demonstrate use > cases. > - Help us get the word out. Are there people, groups, forums or > conferences where we should introduce this capability? > - If you are interested in testing famfs, please do – and let me know > how we can help. > > Micron’s interest is in enabling an ecosystem where shared memory is > practically usable. If famfs is successful, other access methods will > surely emerge. Famfs is our attempt to enable shared memory via an > existing, ubiquitous interface – making it easy to use without having to > adopt new abstractions in advance. > > Thanks for reading, > John Groves > Micron >
