@Jacques, I’m not sure whether past/current literature has a proper term for specific kind of IPC; It would be very helpful to orientate present and future people on the codebase moving forward But we don’t have to be pedantic right now … we can always revisit the issue when the right time approaches…
Thnx Raymond Tay -----Original Message----- From: Jacques Nadeau <jacq...@apache.org> Reply-To: "dev@arrow.apache.org" <dev@arrow.apache.org> Date: Wednesday, 16 March 2016 at 8:54 AM To: "dev@arrow.apache.org" <dev@arrow.apache.org> Subject: Re: Understanding "shared" memory implications >@Corey >The POC Steven and Wes are working on is based on MappedBuffer but I'm >looking at using netty's fork of tcnative to use shared memory directly. > >@Yiannis >We need to have both RPC and a shared memory mechanisms (what I'm inclined >to call IPC but is a specific kind of IPC). The idea is we negotiate via >RPC and then if we determine shared locality, we work over shared memory >(preferably for both data and control). So the system interacting with >HBase in your example would be the one responsible for placing collocated >execution to take advantage of IPC. > >How do others feel of my redefinition of IPC to mean the same memory space >communication (either via shared memory or rdma) versus RPC as socket based >communication? > > >On Tue, Mar 15, 2016 at 5:38 PM, Corey Nolet <cjno...@gmail.com> wrote: > >> I was seeing Netty's unsafe classes being used here, not mapped byte >> buffer not sure if that statement is completely correct but I'll have to >> dog through the code again to figure that out. >> >> The more I was looking at unsafe, it makes sense why that would be >> used.apparently it's also supposed to be included on Java 9 as a first >> class API >> On Mar 15, 2016 7:03 PM, "Wes McKinney" <w...@cloudera.com> wrote: >> >> > My understanding is that you can use java.nio.MappedByteBuffer to work >> > with memory-mapped files as one way to share memory pages between Java >> > (and non-Java) processes without copying. >> > >> > I am hoping that we can reach a POC of zero-copy Arrow memory sharing >> > Java-to-Java and Java-to-C++ in the near future. Indeed this will have >> > huge implications once we get it working end to end (for example, >> > receiving memory from a Java process in Python without a heavy ser-de >> > step -- it's what we've always dreamed of) and with the metadata and >> > shared memory control flow standardized. >> > >> > - Wes >> > >> > On Wed, Mar 9, 2016 at 9:25 PM, Corey J Nolet <cjno...@gmail.com> wrote: >> > > If I understand correctly, Arrow is using Netty underneath which is >> > using Sun's Unsafe API in order to allocate direct byte buffers off heap. >> > It is using Netty to communicate between "client" and "server", >> information >> > about memory addresses for data that is being requested. >> > > >> > > I've never attempted to use the Unsafe API to access off heap memory >> > that has been allocated in one JVM from another JVM but I'm assuming this >> > must be the case in order to claim that the memory is being accessed >> > directly without being copied, correct? >> > > >> > > The implication here is huge. If the memory is being directly shared >> > across processes by them being allowed to directly reach into the direct >> > byte buffers, that's true shared memory. Otherwise, if there's copies >> going >> > on, it's less appealing. >> > > >> > > >> > > Thanks. >> > > >> > > Sent from my iPad >> > >>
