Hmm, somehow I missed those two alternatives, thanks for pointing them out.
I agree that these are probably better than taking a new dependency. Of the two of them, it seems like using Unsafe directly might be better since it would also solve a the issue of setting special environment variables for Netty [1], but it might be two big of a change to couple the two together. The other point brought on the JIRA about honoring -XX:MaxDirectMemorySize is a good one. The one downside to this is it potentially comes with a performance penalty [2] (this is quite dated though). But I think we can always explore other options after doing the simplest thing first. -Micah [1] https://issues.apache.org/jira/browse/ARROW-7223 [2] http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-February/017089.html On Sun, Jan 19, 2020 at 3:03 PM Jacques Nadeau <[email protected]> wrote: > It seems like jna is overkill & unnecessary for simply allocating/freeing > memory. > > A simple way to do this is either to use unsafe directly or call the > existing netty unsafe facade directly. > > PlatformDependent.allocateMemory(long) > PlatformDependent.freeMemory(long) > > Should be relatively straightforward to add to the existing Netty-based > allocator. > > On Sat, Jan 18, 2020 at 8:14 PM Fan Liya <[email protected]> wrote: > >> Hi Micah, >> >> Thanks for the good suggestion. JNA seems like a good and reasonable tool >> for allocating large memory chunks. >> >> How about we directly use Java UNSAFE API? It seems the allocateMemory API >> is also based on the malloc method of the native implementation [1]. >> >> Best, >> Liya Fan >> >> [1] >> >> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/4fc084dac61e/src/share/vm/prims/unsafe.cpp >> >> On Sat, Jan 18, 2020 at 12:58 PM Micah Kornfield <[email protected]> >> wrote: >> >> > With the recently merged changes to the underlying ArrowBuf APIs to >> allow >> > 64-bit memory address spaces there is some follow-up work to actually >> > confirm it works. I opened a JIRA [1] to track this work. >> > >> > The main question is how to provide an allocator that supports larger >> > memory chunks. It appears the Netty API only takes an 32-bit integer >> for >> > array sizes. Doing a little bit of investigation it seems like JNA [2] >> > exposes a direct call to malloc of 64-bit integers [3]. >> > >> > The other options would seem to be rolling our own allocator via JNI. >> > >> > Is there anybody worked with JNA and can share experiences? >> > Is anyone familiar with other options? >> > >> > Thanks, >> > Micah >> > >> > [1] https://issues.apache.org/jira/browse/ARROW-7606 >> > [2] https://github.com/java-native-access/jna >> > [3] >> > >> > >> https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/Native.java#L2265 >> > >> >
