Sounds good, I'll leave it up to you which to implement. Thanks for taking it on.
On Sun, Jan 19, 2020 at 8:47 PM Fan Liya <[email protected]> wrote: > Hi Jacques and Micah, > > Thanks for the fruitful discussion. > > It seems netty based allocator and unsafe based allocator have their > specific advantages. > Maybe we can implement both as independent allocators, to support > different scenarios. > > This should not be difficult, as [1] has laid a solid ground for this. > > Best, > Liya Fan > > [1] https://issues.apache.org/jira/browse/ARROW-7329 > > On Mon, Jan 20, 2020 at 11:38 AM Micah Kornfield <[email protected]> > wrote: > >> Hmm, somehow I missed those two alternatives, thanks for pointing them >> out. >> >> I agree that these are probably better than taking a new dependency. Of >> the two of them, it seems like using Unsafe directly might be better since >> it would also solve a the issue of setting special environment variables >> for Netty [1], but it might be two big of a change to couple the two >> together. >> >> The other point brought on the JIRA about honoring -XX:MaxDirectMemorySize >> is a good one. The one downside to this is it potentially comes with a >> performance penalty [2] (this is quite dated though). But I think we can >> always explore other options after doing the simplest thing first. >> >> -Micah >> >> [1] https://issues.apache.org/jira/browse/ARROW-7223 >> [2] >> >> http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-February/017089.html >> >> On Sun, Jan 19, 2020 at 3:03 PM Jacques Nadeau <[email protected]> >> wrote: >> >> > It seems like jna is overkill & unnecessary for simply >> allocating/freeing >> > memory. >> > >> > A simple way to do this is either to use unsafe directly or call the >> > existing netty unsafe facade directly. >> > >> > PlatformDependent.allocateMemory(long) >> > PlatformDependent.freeMemory(long) >> > >> > Should be relatively straightforward to add to the existing Netty-based >> > allocator. >> > >> > On Sat, Jan 18, 2020 at 8:14 PM Fan Liya <[email protected]> wrote: >> > >> >> Hi Micah, >> >> >> >> Thanks for the good suggestion. JNA seems like a good and reasonable >> tool >> >> for allocating large memory chunks. >> >> >> >> How about we directly use Java UNSAFE API? It seems the allocateMemory >> API >> >> is also based on the malloc method of the native implementation [1]. >> >> >> >> Best, >> >> Liya Fan >> >> >> >> [1] >> >> >> >> >> http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/4fc084dac61e/src/share/vm/prims/unsafe.cpp >> >> >> >> On Sat, Jan 18, 2020 at 12:58 PM Micah Kornfield < >> [email protected]> >> >> wrote: >> >> >> >> > With the recently merged changes to the underlying ArrowBuf APIs to >> >> allow >> >> > 64-bit memory address spaces there is some follow-up work to actually >> >> > confirm it works. I opened a JIRA [1] to track this work. >> >> > >> >> > The main question is how to provide an allocator that supports larger >> >> > memory chunks. It appears the Netty API only takes an 32-bit integer >> >> for >> >> > array sizes. Doing a little bit of investigation it seems like JNA >> [2] >> >> > exposes a direct call to malloc of 64-bit integers [3]. >> >> > >> >> > The other options would seem to be rolling our own allocator via JNI. >> >> > >> >> > Is there anybody worked with JNA and can share experiences? >> >> > Is anyone familiar with other options? >> >> > >> >> > Thanks, >> >> > Micah >> >> > >> >> > [1] https://issues.apache.org/jira/browse/ARROW-7606 >> >> > [2] https://github.com/java-native-access/jna >> >> > [3] >> >> > >> >> > >> >> >> https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/Native.java#L2265 >> >> > >> >> >> > >> >
