Hi Venkat, as far as I understand, arrow works on infrastructures which already support RDMA and it's not included in the core arrow library. Can the arrow developers confirm that assumption?
Thanks On 5 March 2016 at 01:54, Venkat Krishnamurthy <nivik...@gmail.com> wrote: > All > > I've been following along with great interest, and have a n00b question. > > What happens when any of the arrow-capable processing tools needs to work > with a data set or structure that is bigger than a single node's memory > capacity? Does arrow itself handle the distribution of the resulting > columnar representation over multiple nodes > > Or is this the responsibility of the framework/tool/data service that uses > arrow? > > (i know arrow is defined to be a library, but i did see mentions of scatter > gather reads and RDMA, so i'm somewhat confused) > > On the same topic, is there any intent of using a distributed shared memory > like http://grappa.io? the PGAS (partitioned global address space) model > seems like a natural fit for arrow's aspirations > > Thanks > Venkat >
