I don't have deep expertise here, but I think we should not choose one of the options with Netty. There has been a decent amount of work to decouple the Arrow from any hard netty dependencies.
On Mon, Jul 20, 2020 at 3:52 AM Hongze Zhang <notify...@126.com> wrote: > Hi, > > I want to discuss a bit about the discussion[1] in the pending PR[2] for > Java Dataset(it's no longer "Datasets" I guess?) API. > > > - Background: > > We are transferring C++ Arrow buffers to Java side BufferAllocators. We > should decide whether to use -XX:MaxDirectMemorySize as a limit of these > buffers. If yes, what should be a desired solution? > > - Possible alternative solutions so far: > > 1. Reserve from Bits.java from Java side > > Pros: Share memory counter with JVM direct byte buffers, No JNI overhead, > less codes > Cons: More invocations (each buffer a call to Bits#reserveMemory) > > 2. Reserve from Bits.java from C++ side > > Pros: Share memory counter with JVM direct byte buffers, Less invocations > (e.g. if using Jemalloc, we can somehow perform one call for one underlying > trunk) > Cons: JNI overhead, more codes > > 3. Reserve from Netty's PlatformDependent.java from Java side > > Pros: Share memory counter with Netty-based buffers, No JNI overhead, less > codes > Cons: More invocations > > 4. Reserve from Netty's PlatformDependent.java from C++ side > > Pros: Share memory counter with Netty-based buffers, Less invocations > Cons: JNI overhead, more codes > > 5. Not to implement any of the above, respect to BufferAllocator's limit > only. > > > So far I prefer 5, not to use any of the solutions. I am not sure if > "direct memory" is a good indicator for these off-heap buffers, because we > should finally have to decide to share counter with either JVM direct byte > buffers or Netty-based buffers. As far as I could think about, a complete > solution may ideally either to have a global counter for all types of > off-heap buffers, or give each type a individual counter. > > So do you have any thoughts or suggestions on this topic? It would be > great if we could have a conclusion soon as the PR was blocked for some > time. Thanks in advance :) > > > Best, > Hongze > > [1] https://github.com/apache/arrow/pull/7030#issuecomment-657096664 > [2] https://github.com/apache/arrow/pull/7030 >
