Thank you, this is helpful. On the page alignment issue, is it enough to ensure that each buffer (according to the IPC protocol) has a size with a certain multiple? This might be something we can make configurable on the IPC write path on the C++ side. There is nothing fundamentally illegal in the protocol about writing extra padding. Currently generally only pad to 8-byte or 64-byte boundaries.
- Wes On Sat, Mar 30, 2019 at 10:20 AM Dimitris Lekkas <dlekk...@gmail.com> wrote: > > Hi, > > Basically, fletcher has emphasized on reading an Arrow schema and > generating a > hardware interface along with a template for the functional part of the > accelerator. > That way the FPGAs are aware for arrow-backed data. > > On the other hand, we do not work on enabling FPGAs to interpret arrow > formats but > we rather want to feed them with just the "pure" data (I am referring to > arrow columns). > Relying on a generated hardware-interface is a different approach because > on our case > we do not want FPGAs to be aware that Arrow exists. We do not generate any > code > related to the specialized hardware. > > For the above reason, we wanted columns to be page-aligned in order to > enable > fast DMA transfers on them. > > Regards > > > On Fri, Mar 29, 2019 at 6:59 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > hi Dimitris, > > > > Could you comment on how your work is similar to this earlier project > > > > https://github.com/johanpel/fletcher > > > > There's a risk of some fragmentation of efforts if we end up with > > multiple third parties doing this work in isolation; I hope that some > > collaborations are able to take place inside Apache Arrow > > > > - Wes > > > > On Fri, Mar 29, 2019 at 4:27 AM Dimitris Lekkas <dlekk...@gmail.com> > > wrote: > > > > > > Hello, > > > > > > 1) Wes the PR is already submitted and awaiting review. > > > 2) Antoine, you will have feedback from us on the issue shortly. > > > 3) Melik, we plan to contribute back to the Arrow community. > > Particularly, > > > we are working on tweaking plasma to support allocations enabling fast > > > interaction with FPGAs. I will make a proposal for changes pertaining to > > > plasma, so you will hear from me soon. Additionally, we have enabled > > > page alignment of data buffers in Java implementation so if that sounds > > > interesting to you we could discuss on that as well. > > > > > > Regards, > > > > > > Dimitris > > > > > > On Fri, Mar 29, 2019 at 6:31 AM Melik-Adamyan, Areg < > > > areg.melik-adam...@intel.com> wrote: > > > > > > > Hi Chris, > > > > > > > > Do you have plans to contribute the infrastructure part back to the > > > > community so the others can build hybrid pipelines? > > > > > > > > -----Original Message----- > > > > From: Wes McKinney [mailto:wesmck...@gmail.com] > > > > Sent: Thursday, March 28, 2019 10:51 AM > > > > To: dev@arrow.apache.org > > > > Cc: ch...@inaccel.com > > > > Subject: Re: FPGA support for Apache Arrow > > > > > > > > hi Chris -- in addition to commenting on ARROW-2447, do you want to > > submit > > > > a PR to add yourself to Powered By? > > > > > > > > https://github.com/apache/arrow/blob/master/site/powered_by.md > > > > > > > > best > > > > Wes > > > > > > > > On Thu, Mar 28, 2019 at 6:01 AM Antoine Pitrou <anto...@python.org> > > wrote: > > > > > > > > > > > > > > > Hello Chris, > > > > > > > > > > Le 28/03/2019 à 11:56, ch...@inaccel.com a écrit : > > > > > > > > > > > > I would like to let you know that we have released our new version > > > > > > <https://docs.inaccel.com/latest/manager/examples/> of our > > > > > > framework that allows FPGAs to talk to any other framework under > > Arrow. > > > > > > > > > > > > That means that any Arrow-based framework can utilize now FPGAs in > > > > > > order to speedup their applications (e.g. ML, analytics, > > compression, > > > > etc.). > > > > > > > > > > Since you are doing heterogenous computing with Arrow, it would be > > > > > extra nice if you / your engineers could take some time to give > > > > > feedback on this issue: > > > > > https://issues.apache.org/jira/browse/ARROW-2447 > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > >
