RE: [DISCUSS] Adding new columnar memory layouts to Arrow (in-memory, IPC, C ABI)

2021-12-16 Thread Yang, Binwei
"1. Is it so those projects could use the arrow compute kernels instead of their own?" This is the case we met. We offloaded Spark JVM SQL engine to native library (gazelle-plugin) which is based on arrow computer kernel, but customer asked if it's possible to use Velox as native backend becaus

Re: [RESULT][VOTE] Proposed addition to Arrow Flight: Arrow Flight RPC

2021-12-16 Thread David Li
Strictly speaking we should have a vote since it is updating the format definition files we already voted on. I am curious about what exactly you mean by additional column metadata, but if it's just going to be encoded into the key-value metadata then I don't see a problem there. (As in: it so

Re: [RESULT][VOTE] Proposed addition to Arrow Flight: Arrow Flight RPC

2021-12-16 Thread James Duong
Hi David, While working on the JDBC driver on top of Flight SQL and on integration tests, we identified a couple of enhancements that were needed. 1. The ability to report data type information, as done in this PR: https://github.com/apache/arrow/pull/11982. This PR adds another RPC request for th

Re: [DISCUSS] Splitting the sphinx-based Arrow docs into separate sphinx projects

2021-12-16 Thread Joris Van den Bossche
I have a draft PR experimenting with this at https://github.com/apache/arrow/pull/11980 On Wed, 8 Dec 2021 at 17:17, Antoine Pitrou wrote: > > Hi Joris, > > I have a question: does Intersphinx handle glossary links as well? > > Good question. I didn't try explicitly (will do), but I did update o

Re: [RESULT][VOTE] Proposed addition to Arrow Flight: Arrow Flight RPC

2021-12-16 Thread Kyle Porter
Thanks All - we'll look to get the tests merged into this branch so we can close ASAP. *Kyle Porter* CEO Bit Quill Technologies Inc. Office: +1.778.331.3355 | Direct: +1.604.441.7318 | ky...@bitquilltech.com https://www.bitquill.com This email message is for the sole use of the intended recipient

Re: [DISCUSS] Adding new columnar memory layouts to Arrow (in-memory, IPC, C ABI)

2021-12-16 Thread Andrew Lamb
> DuckDB and Velox are two projects which have designed themselves to be > very nearly Arrow-compatible but have implemented alternative memory > layouts to achieve O(# records) selections on all data types. I am > proposing to adopt these innovations as additional memory layouts in > Arrow with a