Hi Yurui, > Let’s say passed memory addresses from c++ to JVM and constructed the data > structure in Java.
I think that you should add a method to release off heap the memory area on the native side and call it on the Java side in this case. In Gandiva, it seems to allocate the memory area of return values on the Java side, pass its pointer to native. https://github.com/dremio/gandiva/blob/master/java/src/test/java/org/apache/arrow/gandiva/evaluator/NativeEvaluatorTest.java#L142 https://github.com/dremio/gandiva/blob/master/java/src/main/java/org/apache/arrow/gandiva/evaluator/NativeEvaluator.java#L152 And release it on Java side. https://github.com/dremio/gandiva/blob/master/java/src/test/java/org/apache/arrow/gandiva/evaluator/NativeEvaluatorTest.java#L158 > how could I make sure the memory will be released when necessary? I think you need to use the memory diagnostic tool. https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks005.html thanks. 2018年7月20日(金) 9:11 Wes McKinney <wesmck...@gmail.com>: > > hi Yurui, > > I don't know if anyone has worked out a way to permanently transfer > ownership from memory allocated by the Java or C++ libraries. This is > partially what Plasma is for. I am not sure how Gandiva is currently > dealing with memory management between Java and C++ in-process. If > someone wants to comment that would be helpful. > > - Wes > > On Tue, Jul 17, 2018 at 12:19 AM, 周宇睿(闻拙) <yurui....@alibaba-inc.com> wrote: > > Hi Wes: > > > > Thank you for the response. Yes the examples you provided are very helpful. > > > > But I still have a question regarding memory management. Let’s say passed > > memory addresses from c++ to JVM and constructed the data structure in Java. > > Since this is an off heap memory, how could I make sure the memory will be > > released when necessary? > > > > thanks > > Yurui > > > > from Alimail macOS > > > > ------------------Original Mail ------------------ > > Sender:Wes McKinney <wesmck...@gmail.com> > > Send Date:Tue Jul 17 02:09:51 2018 > > Recipients: <dev@arrow.apache.org> > > Subject:Re: Passing Arrow object across language > >> > >> I discussed some of these things at a high level in my talk at SciPy > >> 2018 last week > >> > >> > >> https://www.slideshare.net/wesm/apache-arrow-crosslanguage-development-platform-for-inmemory-data-105427919 > >> > >> On Mon, Jul 16, 2018 at 2:08 PM, Wes McKinney <wesmck...@gmail.com> wrote: > >> > hi Yurui, > >> > > >> > You can also share data structures through JNI without using the IPC > >> > tools at all, which could require memory copying to produce the IPC > >> > messages. > >> > > >> > What you can do is obtain the memory addresses for the component > >> > buffers of an array (or vector, as called in Java) and construct the > >> > data structure from the memory addresses on the other side. We are > >> > doing exactly this already in Python using JPype (which is JNI-based): > >> > > >> > https://github.com/apache/arrow/blob/master/python/pyarrow/jvm.py > >> > > >> > The Gandiva project uses JNI to pass Java Netty buffer memory > >> > addresses to C++, you can see the code for creating the arrays from > >> > the memory addresses and then constructing a RecordBatch: > >> > > >> > > >> > https://github.com/dremio/gandiva/blob/master/cpp/src/jni/native_builder.cc#L602 > >> > > >> > I believe as time goes on we will have better and more standardized > >> > APIs to deal with JNI<->C++ zero-copy passing, these implementations > >> > have only been done relatively recently. Your contributions to the > >> > Arrow project around this would be most welcomed! > >> > > >> > Thanks, > >> > Wes > >> > > >> > On Mon, Jul 16, 2018 at 2:00 PM, Philipp Moritz <pcmor...@gmail.com> > >> > wrote: > >> >> Hey Yuri, > >> >> > >> >> you can use the Arrow IPC mechanism to do this: > >> >> > >> >> - https://github.com/apache/arrow/blob/master/format/IPC.md > >> >> - Python: https://arrow.apache.org/docs/python/ipc.html > >> >> - C++: https://arrow.apache.org/docs/cpp/namespacearrow_1_1ipc.html > >> >> - For Java, see the org.apache.arrow.vector.ipc namespace > >> >> > >> >> On the C++ side, you can for example use a RecordBatchStreamWriter to > >> >> write > >> >> the IPC message, and then on the Java side you could use the > >> >> ArrowStreamReader to read it. > >> >> > >> >> There are some tests here: > >> >> > >> >> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-write-test.cc > >> >> > >> >> https://github.com/apache/arrow/tree/master/java/vector/src/test/java/org/apache/arrow/vector/ipc > >> >> > >> >> There is also integration tests here, although I'm not really familiar > >> >> with > >> >> them: > >> >> > >> >> https://github.com/apache/arrow/tree/master/integration > >> >> > >> >> If you could write a little tutorial/into on how to do this (maybe > >> >> using > >> >> Plasma for exchanging the data) and contribute it to the documentation, > >> >> that would be amazing! > >> >> > >> >> Best, > >> >> Philipp. > >> >> > >> >> On Mon, Jul 16, 2018 at 4:14 AM, 周宇睿(闻拙) <yurui....@alibaba-inc.com> > >> >> wrote: > >> >> > >> >>> Hi guys: > >> >>> > >> >>> I might miss something quite obviously. But how does Arrow passing > >> >>> objects > >> >>> across language? Let’s say I have a java program that invoke a c++ > >> >>> function > >> >>> via JNI, how does the c++ function pass an Arrow RecordBack object > >> >>> back to > >> >>> Java runtime without memory copy? > >> >>> > >> >>> Any advise would be appreciated. > >> >>> Thanks > >> >>> Yurui > >> >>> > >> >>> from Alimail macOS -- 高橋 真之
