Here's the relevant bit from my article<http://www.zdnet.com/article/apache-arrow-unifies-in-memory-big-data-systems/>, based on my discussion with Jaques:
Partying on the data Once multiple projects adopt Arrow, they will be able to share data with little overhead, since the data won't need to be serialized and deserialized between multiple proprietary, in-memory data formats. For systems installed on the same cluster, sharing the memory on each node, the data need not move nor transform at all. Instead, different projects and products can pipeline together, each taking turns working with the data, in a cumulative fashion. -----Original Message----- From: Michael D. Coon [mailto:mdco...@yahoo.com.INVALID] Sent: Wednesday, February 24, 2016 2:21 PM To: dev@arrow.apache.org Subject: Re: Question about mutability I had the same understanding as Corey and thought that apps shared an allocated memory space for the sole purpose of eliminating the need to copy data between the apps. If it's just a replacement for protobuf SERDE, that makes it a whole lot less exciting :( On Wednesday, February 24, 2016 2:17 PM, Andrew Brust <andrew.br...@bluebadgeinsights.com<mailto:andrew.br...@bluebadgeinsights.com>> wrote: Hmm...that's not exactly how Jaques described things to me when he briefed me on Arrow ahead of the announcement. -----Original Message----- From: Zhe Zhang [mailto:z...@apache.org] Sent: Wednesday, February 24, 2016 2:08 PM To: dev@arrow.apache.org<mailto:dev@arrow.apache.org> Subject: Re: Question about mutability I don't think one application/process's memory space will be made available to other applications/processes. It's fundamentally hard for processes to share their address spaces. IIUC, with Arrow, when application A shares data with application B, the data is still duplicated in the memory spaces of A and B. It's just that data serialization/deserialization are much faster with Arrow (compared with Protobuf). On Wed, Feb 24, 2016 at 10:40 AM Corey Nolet <cjno...@gmail.com<mailto:cjno...@gmail.com>> wrote: > Forgive me if this question seems ill-informed. I just started looking > at Arrow yesterday. I looked around the github a tad. > > Are you expecting the memory space held by one application to be > mutable by that application and made available to all applications > trying to read the memory space? >
