hi Yurui, I don't know if anyone has worked out a way to permanently transfer ownership from memory allocated by the Java or C++ libraries. This is partially what Plasma is for. I am not sure how Gandiva is currently dealing with memory management between Java and C++ in-process. If someone wants to comment that would be helpful.
- Wes On Tue, Jul 17, 2018 at 12:19 AM, 周宇睿(闻拙) <yurui....@alibaba-inc.com> wrote: > Hi Wes: > > Thank you for the response. Yes the examples you provided are very helpful. > > But I still have a question regarding memory management. Let’s say passed > memory addresses from c++ to JVM and constructed the data structure in Java. > Since this is an off heap memory, how could I make sure the memory will be > released when necessary? > > thanks > Yurui > > from Alimail macOS > > ------------------Original Mail ------------------ > Sender:Wes McKinney <wesmck...@gmail.com> > Send Date:Tue Jul 17 02:09:51 2018 > Recipients: <dev@arrow.apache.org> > Subject:Re: Passing Arrow object across language >> >> I discussed some of these things at a high level in my talk at SciPy >> 2018 last week >> >> >> https://www.slideshare.net/wesm/apache-arrow-crosslanguage-development-platform-for-inmemory-data-105427919 >> >> On Mon, Jul 16, 2018 at 2:08 PM, Wes McKinney <wesmck...@gmail.com> wrote: >> > hi Yurui, >> > >> > You can also share data structures through JNI without using the IPC >> > tools at all, which could require memory copying to produce the IPC >> > messages. >> > >> > What you can do is obtain the memory addresses for the component >> > buffers of an array (or vector, as called in Java) and construct the >> > data structure from the memory addresses on the other side. We are >> > doing exactly this already in Python using JPype (which is JNI-based): >> > >> > https://github.com/apache/arrow/blob/master/python/pyarrow/jvm.py >> > >> > The Gandiva project uses JNI to pass Java Netty buffer memory >> > addresses to C++, you can see the code for creating the arrays from >> > the memory addresses and then constructing a RecordBatch: >> > >> > >> > https://github.com/dremio/gandiva/blob/master/cpp/src/jni/native_builder.cc#L602 >> > >> > I believe as time goes on we will have better and more standardized >> > APIs to deal with JNI<->C++ zero-copy passing, these implementations >> > have only been done relatively recently. Your contributions to the >> > Arrow project around this would be most welcomed! >> > >> > Thanks, >> > Wes >> > >> > On Mon, Jul 16, 2018 at 2:00 PM, Philipp Moritz <pcmor...@gmail.com> >> > wrote: >> >> Hey Yuri, >> >> >> >> you can use the Arrow IPC mechanism to do this: >> >> >> >> - https://github.com/apache/arrow/blob/master/format/IPC.md >> >> - Python: https://arrow.apache.org/docs/python/ipc.html >> >> - C++: https://arrow.apache.org/docs/cpp/namespacearrow_1_1ipc.html >> >> - For Java, see the org.apache.arrow.vector.ipc namespace >> >> >> >> On the C++ side, you can for example use a RecordBatchStreamWriter to >> >> write >> >> the IPC message, and then on the Java side you could use the >> >> ArrowStreamReader to read it. >> >> >> >> There are some tests here: >> >> >> >> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-write-test.cc >> >> >> >> https://github.com/apache/arrow/tree/master/java/vector/src/test/java/org/apache/arrow/vector/ipc >> >> >> >> There is also integration tests here, although I'm not really familiar >> >> with >> >> them: >> >> >> >> https://github.com/apache/arrow/tree/master/integration >> >> >> >> If you could write a little tutorial/into on how to do this (maybe >> >> using >> >> Plasma for exchanging the data) and contribute it to the documentation, >> >> that would be amazing! >> >> >> >> Best, >> >> Philipp. >> >> >> >> On Mon, Jul 16, 2018 at 4:14 AM, 周宇睿(闻拙) <yurui....@alibaba-inc.com> >> >> wrote: >> >> >> >>> Hi guys: >> >>> >> >>> I might miss something quite obviously. But how does Arrow passing >> >>> objects >> >>> across language? Let’s say I have a java program that invoke a c++ >> >>> function >> >>> via JNI, how does the c++ function pass an Arrow RecordBack object >> >>> back to >> >>> Java runtime without memory copy? >> >>> >> >>> Any advise would be appreciated. >> >>> Thanks >> >>> Yurui >> >>> >> >>> from Alimail macOS
