Ahh, sorry that was in the title. I think this might become easier once https://issues.apache.org/jira/browse/ARROW-3191 is complete, which I think is being actively worked on, but others who are more familiar with the Java implementation can chime in.
On Wednesday, March 20, 2019, Dimitris Lekkas <dlekk...@gmail.com> wrote: > This refers to the Java implementation. Our backend is primarily written > in Java, > thus we decided to prototype our integration in Java. To elucidate, let's > assume > that we want to create a shared file in memory (e.g /dev/shm) and FPGAs > will > later DMA this shared memory area. By utilizing the current Java > implementation > to populate an Arrow schema, the addresses where data buffers of > BaseValueVectors > reside are not page-aligned. Therefore, we wanted those addresses to be > page-aligned, > so we could natively mmap() such addresses to shared-files in /dev/shm. > Alternatively, > we could also benefit from being able to leverage MemoryMappedFiles to > data buffers. > > Dimitris > > On Wed, Mar 20, 2019 at 11:17 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> Hi Dimitris, >> This sounds interesting. Is this for the C++ implementation? Could you >> go >> into a little bit more detail? How would this differ then using a >> MemoryPool implementation that always aligns to 4KB boundaries (instead of >> the 64 byte boundaries the default one does today [1])? >> >> Thanks, >> Micah >> >> [1] >> https://github.com/apache/arrow/blob/master/cpp/src/ >> arrow/memory_pool.cc#L40 >> >> On Wed, Mar 20, 2019 at 1:53 AM Dimitris Lekkas <dlekk...@gmail.com> >> wrote: >> >> > Hello folks, >> > >> > I am working at Inaccel where we utilize FPGAs to accelerate machine >> > learning workloads . Recently, we wanted to integrate our platform with >> > Arrow and we stumbled upon the non-alignment of data-buffers to page >> > boundaries (4KB). We implemented the option to supply per-column >> metadata >> > to page-align column vectors and we later memory mapped those columns >> > using native mmap() calls. Accelerators leverage page-alignment to i/o >> > efficiently, thus such an option might be an interesting addition to the >> > project. >> > In case you are interested, I will create a PR. >> > >> > Dimitris >> > >> >
