> I don't really grok how Arrow would be used with Avatica now (I struggle > with how Arrow would be used in place of Protobuf, period),
For the packets that carry rows I would envision a payload format where there rows are just a byte-array. The default payload format would remain protobuf. The rest of the packet would remain protobuf. An arrow payload would still be, in a sense, protobuf, since one of the value types in protobuf (I assume) is a byte array. Other packet types would remain 100% protobuf. This only matters for "large" data. To achieve zero copy or single-copy, we'd have to devise a way to have the producer write into the under-construction protobuf packet in situ, or perhaps have the network adapter construct a packet from fragments in several locations. Julian On Fri, Aug 17, 2018 at 11:33 AM Enrico Olivelli <[email protected]> wrote: > > I am interested in this topic too. > I am not (yet?) using Avatica, but I am working a lot in order to save > resources during data access and data transfer in JDBC driver in my project > HerdDB (which is using Calcite SQLParser and planner). > > Currently the only way it is have a proprietary protocol and plumb deeply > the network/RPC layer to the data access/internal representation layer. > > Joining the efforts and implementing an efficient stack will be great > > I will follow the discussion, if possible I will try to contribute. > > Enrico > > Il ven 17 ago 2018, 18:05 Josh Elser <[email protected]> ha scritto: > > > Thanks for the heads-up, Julian. Subscribing. > > > > I don't really grok how Arrow would be used with Avatica now (I struggle > > with how Arrow would be used in place of Protobuf, period), but I should > > make the time to figure that out. > > > > On 8/16/18 1:27 PM, Julian Hyde wrote: > > > There's a discussion on the Apache Arrow list about a proposed RPC > > > mechanism called Flight. > > > > > > One of its use cases is for executing SQL. I fear that for SQL they > > > will likely end up with a protocol similar to Avatica. I think that we > > > should extend Avatica should allow Arrow as its data format, with as > > > few copies on server and client side as possible. I would welcome > > > contributions from the Arrow community. > > > > > > Here is my email: > > > > > https://lists.apache.org/thread.html/d02133aee410d68521165ee3fcfd8f395fb1e6ed630af8df96c15397@%3Cdev.arrow.apache.org%3E > > > > > > Please join that thread if you are interested. > > > > > > Julian > > > > > > -- > > > -- Enrico Olivelli
