The Arrow Flight blog post [1] was published today I think it looks quite good -- thank you to everyone who helped make that happen;
There is a discussion ticket[1] for possibly making a Rust implementation if anyone looking for a neat project [1]: https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/ [2]: https://github.com/apache/arrow-rs/issues/1323 On Tue, Feb 8, 2022 at 4:14 PM David Li <lidav...@apache.org> wrote: > Thanks for the clarification, Wes! > > Kyle - the grant process is outlined here [1] and I can help with this on > the Arrow PMC side. From your side, you will need to file a grant (either > the CCLA form or the grant here [2]) and make sure everyone has a CLA on > file, then once the Apache side has acknowledged everything we can merge. > > We could hold the Arrow-side vote now and start the process, aiming to > merge into a development branch, if that is more convenient for the > developers, or we can continue iterating on the PR for now. > > There's an example of the process for Arrow/Julia [3]. > > [1]: https://incubator.apache.org/ip-clearance/ip-clearance-template.html > [2]: https://www.apache.org/licenses/contributor-agreements.html > [3]: https://incubator.apache.org/ip-clearance/arrow-julia-library2.html > > -David > > On Fri, Feb 4, 2022, at 18:10, Wes McKinney wrote: > > hi David, > > > > Yes, I think we need to do an IP clearance for this work. Please let > > me know if I can assist, but it would probably be good for other PMC > > members to familiarize themselves with the process since we are likely > > to receive more large pieces of work that need to go through the > > process in the future! > > > > Thanks, > > Wes > > > > On Wed, Jan 26, 2022 at 12:23 PM David Li <lidav...@apache.org> wrote: > >> > >> I'd also like to highlight this new PR which contributes a JDBC driver > on top of Flight SQL and Avatica: > https://github.com/apache/arrow/pull/12254 > >> > >> One thing I'm not sure of is whether this needs to go through IP > clearance? At ~15k LOC and with development going back to June 2021, it is > quite substantial. > >> > >> -David > >> > >> On Mon, Jan 24, 2022, at 14:14, David Li wrote: > >> > Following up here, I think we've resolved all current comments, so if > anyone else has feedback, it would be much appreciated. Otherwise, I think > it would be good to put it to a vote soon, and we can use the 8.0 cycle to > improve the documentation and see if there's any other work needed for the > JDBC driver. > >> > > >> > -David > >> > > >> > On Fri, Jan 21, 2022, at 09:09, David Li wrote: > >> > > Following up here, James Duong and Jose Almeida have submitted a > set of pull requests proposing a set of additions to Flight SQL to expose > more information about supported data types and provide metadata about > column types in results. For anyone interested in reviewing the proposals, > the pull requests can be found here: > >> > > * https://github.com/apache/arrow/pull/11982 > >> > > * https://github.com/apache/arrow/pull/11999 > >> > > These PRs include implementations for C++ and Java as well as > integration tests. > >> > > > >> > > Thanks, > >> > > David > >> > > > >> > > On Fri, Dec 17, 2021, at 17:07, James Duong wrote: > >> > > > Yes, additional metadata would just be using the Field metadata > map. The > >> > > > protocol is the same, we have just pre-defined keys for some > fields that > >> > > > would be used for JDBC column attributes. > >> > > > > >> > > > Our preference would be that we get the currently approved > protocol merged > >> > > > into master first (after completing the integration tests) and > then have a > >> > > > separate vote on the TypeInfo changes. There's significant value > in adding > >> > > > Flight-SQL already and it'd be great to make that available. It's > natural > >> > > > that there will be an ongoing need to add extensions to the > protocol as it > >> > > > gets used in more scenarios. Now that we have a solid foundation, > we can > >> > > > examine further changes on a case-by-case basis. > >> > > > > >> > > > On Thu, Dec 16, 2021 at 2:42 PM David Li <lidav...@apache.org> > wrote: > >> > > > > >> > > > > Strictly speaking we should have a vote since it is updating > the format > >> > > > > definition files we already voted on. > >> > > > > > >> > > > > I am curious about what exactly you mean by additional column > metadata, > >> > > > > but if it's just going to be encoded into the key-value > metadata then I > >> > > > > don't see a problem there. (As in: it sounds like it fits in > the Field > >> > > > > class given it's encoded in the Field metadata!) > >> > > > > > >> > > > > -David > >> > > > > > >> > > > > On Thu, Dec 16, 2021, at 16:14, James Duong wrote: > >> > > > > > Hi David, > >> > > > > > > >> > > > > > While working on the JDBC driver on top of Flight SQL and on > integration > >> > > > > > tests, we identified a couple of enhancements that were > needed. > >> > > > > > 1. The ability to report data type information, as done in > this PR: > >> > > > > > https://github.com/apache/arrow/pull/11982. This PR adds > another RPC > >> > > > > > request for this information. > >> > > > > > 2. Additional column metadata that's outside of the > Schema/Field classes > >> > > > > in > >> > > > > > Arrow (PR pending) when returning Arrow schemas. The planned > PR uses the > >> > > > > > Arrow Field's MetadataMap to encode extra metadata rather > than altering > >> > > > > any > >> > > > > > protobuf definitions. > >> > > > > > > >> > > > > > Should these additional changes go in together with the rest > of > >> > > > > Flight-SQL, > >> > > > > > or be approved separately? > >> > > > > > > >> > > > > > On Thu, Dec 16, 2021 at 7:54 AM Kyle Porter < > ky...@bitquilltech.com > >> > > > > .invalid> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Thanks All - we'll look to get the tests merged into this > branch so we > >> > > > > can > >> > > > > > > close ASAP. > >> > > > > > > > >> > > > > > > *Kyle Porter* > >> > > > > > > CEO > >> > > > > > > Bit Quill Technologies Inc. > >> > > > > > > Office: +1.778.331.3355 | Direct: +1.604.441.7318 | > >> > > > > ky...@bitquilltech.com > >> > > > > > > https://www.bitquill.com > >> > > > > > > > >> > > > > > > This email message is for the sole use of the intended > recipient(s) > >> > > > > and may > >> > > > > > > contain confidential and privileged information. Any > unauthorized > >> > > > > review, > >> > > > > > > use, disclosure, or distribution is prohibited. If you are > not the > >> > > > > > > intended recipient, please contact the sender by reply > email and > >> > > > > destroy > >> > > > > > > all copies of the original message. Thank you. > >> > > > > > > > >> > > > > > > > >> > > > > > > On Wed, Dec 15, 2021 at 9:11 AM David Li < > lidav...@apache.org> wrote: > >> > > > > > > > >> > > > > > > > My vote: +1 > >> > > > > > > > > >> > > > > > > > The vote passes with three +1 (binding) votes, one +1 > (non binding) > >> > > > > vote, > >> > > > > > > > and one -0.5 (binding) vote. > >> > > > > > > > > >> > > > > > > > However, we will first merge into a separate branch and > implement > >> > > > > > > > integration tests before merging into the main branch. > JIRA for > >> > > > > > > integration > >> > > > > > > > tests: https://issues.apache.org/jira/browse/ARROW-15112 > >> > > > > > > > > >> > > > > > > > @Kyle I've created the branch flight-sql[1], would you > prefer I > >> > > > > merge in > >> > > > > > > > your existing PRs, or would you prefer to create new PRs > against that > >> > > > > > > > branch (given you've already started on things)? > >> > > > > > > > > >> > > > > > > > On a side note - do we document the requirements for > proposed > >> > > > > additions > >> > > > > > > > somewhere? (multiple implementations, integration tests) > It would be > >> > > > > nice > >> > > > > > > > to have it on hand for reference. > >> > > > > > > > > >> > > > > > > > [1]: https://github.com/apache/arrow/tree/flight-sql > >> > > > > > > > > >> > > > > > > > -David > >> > > > > > > > > >> > > > > > > > On Mon, Dec 13, 2021, at 11:25, Kyle Porter wrote: > >> > > > > > > > > Thanks David, > >> > > > > > > > > > >> > > > > > > > > Yes, the team is actually already looking at adding the > cross > >> > > > > language > >> > > > > > > > > tests apologies for not communicating that earlier > >> > > > > > > > > > >> > > > > > > > > On Mon., Dec. 13, 2021, 12:18 p.m. David Li, < > lidav...@apache.org> > >> > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Are any other PMC members able to look at this? > >> > > > > > > > > > > >> > > > > > > > > > > > > OK by me. We could also create a branch to > merge the PRs > >> > > > > add > >> > > > > > > the > >> > > > > > > > > > > > > integration tests, and then merge all at once. > >> > > > > > > > > > > >> > > > > > > > > > Kyle, is this an ok solution? Would you & your team > be able to > >> > > > > get > >> > > > > > > > > > integration tests done reasonably soon? > >> > > > > > > > > > > >> > > > > > > > > > There's some setup for Flight integration tests > already: > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > https://github.com/apache/arrow/blob/11be9c542b9699b7eb4ae1656775c9b30639e415/dev/archery/archery/integration/runner.py#L375-L385 > >> > > > > > > > > > > >> > > > > > > > > > So what would be needed are: > >> > > > > > > > > > > >> > > > > > > > > > 1. Enable Flight SQL for the integration test > container > >> > > > > > > > > > 2. Link the integration test client/server to Flight > SQL > >> > > > > > > > > > 3. Add one or more test scenarios in the integration > test > >> > > > > runner, and > >> > > > > > > > in > >> > > > > > > > > > the integration test client/server > >> > > > > > > > > > > >> > > > > > > > > > It might be acceptable to just hardcode expected > >> > > > > requests/responses > >> > > > > > > > > > instead of integrating SQLite/Derby (as was done for > the > >> > > > > individual > >> > > > > > > > > > language tests) since the focus should be on just the > protocol > >> > > > > and > >> > > > > > > not > >> > > > > > > > > > particular implementations. > >> > > > > > > > > > > >> > > > > > > > > > -David > >> > > > > > > > > > > >> > > > > > > > > > On Sun, Dec 12, 2021, at 16:21, Wes McKinney wrote: > >> > > > > > > > > > > +1. Agree re: adding integration tests as soon as > practical > >> > > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 9, 2021 at 5:21 AM Ravindra Pindikura < > >> > > > > > > > ravin...@dremio.com> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > +1 > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Wed, Dec 8, 2021 at 11:42 PM Micah Kornfield < > >> > > > > > > > emkornfi...@gmail.com > >> > > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Given that the C++ and Java components are in > separate > >> > > > > PRs, > >> > > > > > > > would > >> > > > > > > > > > it be > >> > > > > > > > > > > > > > acceptable to add after the initial merge? > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > OK by me. We could also create a branch to > merge the PRs > >> > > > > add > >> > > > > > > the > >> > > > > > > > > > > > > integration tests, and then merge all at once. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Wed, Dec 8, 2021 at 10:07 AM Kyle Porter < > >> > > > > > > > ky...@bitquilltech.com> > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Given that the C++ and Java components are in > separate > >> > > > > PRs, > >> > > > > > > > would > >> > > > > > > > > > it be > >> > > > > > > > > > > > > > acceptable to add after the initial merge? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > *Kyle Porter* > >> > > > > > > > > > > > > > CEO > >> > > > > > > > > > > > > > Bit Quill Technologies Inc. > >> > > > > > > > > > > > > > Office: +1.778.331.3355 | Direct: > +1.604.441.7318 | > >> > > > > > > > > > > > > ky...@bitquilltech.com > >> > > > > > > > > > > > > > https://www.bitquill.com > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > This email message is for the sole use of the > intended > >> > > > > > > > > > recipient(s) and > >> > > > > > > > > > > > > > may contain confidential and privileged > information. Any > >> > > > > > > > > > unauthorized > >> > > > > > > > > > > > > > review, use, disclosure, or distribution is > prohibited. > >> > > > > If > >> > > > > > > you > >> > > > > > > > > > are not > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > intended recipient, please contact the sender > by reply > >> > > > > email > >> > > > > > > > and > >> > > > > > > > > > destroy > >> > > > > > > > > > > > > > all copies of the original message. Thank > you. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Wed, Dec 8, 2021 at 2:03 PM Micah > Kornfield < > >> > > > > > > > > > emkornfi...@gmail.com> > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > There is not an integration test. Do we > want to > >> > > > > require > >> > > > > > > > this? > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> It would be nice, I'm -0.5 vote without > one. So if > >> > > > > enough > >> > > > > > > > PMC > >> > > > > > > > > > members > >> > > > > > > > > > > > > >> want to forgo the integration test the vote > can still > >> > > > > pass. > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > Is cross language testing something that's > usually > >> > > > > done? > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> Yes. One of the value propositions of Arrow > is the > >> > > > > > > > cross-language > >> > > > > > > > > > > > > >> support. The community agreed to > specification changes > >> > > > > > > (and I > >> > > > > > > > > > assumed > >> > > > > > > > > > > > > >> this > >> > > > > > > > > > > > > >> covers new specifications) need to have > reference > >> > > > > > > > implementations > >> > > > > > > > > > in > >> > > > > > > > > > > > > >> C++/Java with integration testing between > the two. > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> On Wed, Dec 8, 2021 at 5:21 AM Kyle Porter < > >> > > > > > > > > > ky...@bitquilltech.com > >> > > > > > > > > > > > > >> .invalid> > >> > > > > > > > > > > > > >> wrote: > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > The team initially developed the C++ > client against > >> > > > > the > >> > > > > > > Java > >> > > > > > > > > > server, > >> > > > > > > > > > > > > and > >> > > > > > > > > > > > > >> > have done some cross language testing. It > wasn't > >> > > > > > > exhaustive > >> > > > > > > > or > >> > > > > > > > > > > > > >> methodical > >> > > > > > > > > > > > > >> > in nature, however. Is cross language > testing > >> > > > > something > >> > > > > > > > that's > >> > > > > > > > > > usually > >> > > > > > > > > > > > > >> > done? > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > On Wed., Dec. 8, 2021, 9:18 a.m. David Li, > < > >> > > > > > > > lidav...@apache.org > >> > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > There is not an integration test. Do we > want to > >> > > > > require > >> > > > > > > > this? > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > Also CC @Kyle, in case your team has > done such > >> > > > > testing. > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > It looks like Flight itself did not have > a test for > >> > > > > a > >> > > > > > > few > >> > > > > > > > > > versions > >> > > > > > > > > > > > > >> after > >> > > > > > > > > > > > > >> > > it was initially implemented. > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > -David > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > On Tue, Dec 7, 2021, at 23:19, Micah > Kornfield > >> > > > > wrote: > >> > > > > > > > > > > > > >> > > > Is there an integration test between > the two > >> > > > > > > languages? > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > On Tue, Dec 7, 2021 at 1:35 PM David > Li < > >> > > > > > > > > > lidav...@apache.org> > >> > > > > > > > > > > > > >> wrote: > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > Hello, > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > Kyle Porter, Rafael Telles, Ryan > Nicholson, et. > >> > > > > al. > >> > > > > > > > have > >> > > > > > > > > > > > > proposed > >> > > > > > > > > > > > > >> > > adding > >> > > > > > > > > > > > > >> > > > > Arrow Flight SQL, an experimental > protocol for > >> > > > > > > > > > interacting with > >> > > > > > > > > > > > > >> SQL > >> > > > > > > > > > > > > >> > > > > databases over Arrow Flight [1], as > explained > >> > > > > in a > >> > > > > > > > > > previous ML > >> > > > > > > > > > > > > >> > > discussion > >> > > > > > > > > > > > > >> > > > > [2] and in a design document [3]. > The purpose of > >> > > > > > > > Flight > >> > > > > > > > > > SQL is > >> > > > > > > > > > > > > to > >> > > > > > > > > > > > > >> > allow > >> > > > > > > > > > > > > >> > > > > clients and SQL database servers to > communicate > >> > > > > > > > (execute > >> > > > > > > > > > > > > queries, > >> > > > > > > > > > > > > >> > list > >> > > > > > > > > > > > > >> > > > > tables, create prepared statements, > etc.) using > >> > > > > > > Arrow > >> > > > > > > > and > >> > > > > > > > > > Arrow > >> > > > > > > > > > > > > >> > > Flight, by > >> > > > > > > > > > > > > >> > > > > defining how to use Flight RPC > methods, as well > >> > > > > as > >> > > > > > > > message > >> > > > > > > > > > > > > >> payloads > >> > > > > > > > > > > > > >> > to > >> > > > > > > > > > > > > >> > > use > >> > > > > > > > > > > > > >> > > > > with those methods. > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > The new protocol definitions can be > found at > >> > > > > [4]. > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > They have provided pull requests > implementing > >> > > > > the > >> > > > > > > > server > >> > > > > > > > > > and > >> > > > > > > > > > > > > >> client > >> > > > > > > > > > > > > >> > > > > protocol in C++ [5] and Java [6] > which can be > >> > > > > merged > >> > > > > > > > > > after this > >> > > > > > > > > > > > > >> > > addition is > >> > > > > > > > > > > > > >> > > > > approved. > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > Please vote whether to accept this > addition. The > >> > > > > > > vote > >> > > > > > > > > > will be > >> > > > > > > > > > > > > open > >> > > > > > > > > > > > > >> > for > >> > > > > > > > > > > > > >> > > at > >> > > > > > > > > > > > > >> > > > > least 72 hours. > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > [1]: > >> > > > > > > https://arrow.apache.org/docs/format/Flight.html > >> > > > > > > > > > > > > >> > > > > [2]: > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > https://lists.apache.org/thread/s08b20ty756qq10zybd9qr0mm4jhmz93 > >> > > > > > > > > > > > > >> > > > > [3]: > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing > >> > > > > > > > > > > > > >> > > > > Note that the protocol definitions > in the design > >> > > > > > > > document > >> > > > > > > > > > are > >> > > > > > > > > > > > > out > >> > > > > > > > > > > > > >> of > >> > > > > > > > > > > > > >> > > date; > >> > > > > > > > > > > > > >> > > > > the canonical reference is in the > pull requests. > >> > > > > > > > > > > > > >> > > > > [4]: > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > https://github.com/apache/arrow/blob/72ce72ba855909052f7dfb898105b419697157c8/format/FlightSql.proto > >> > > > > > > > > > > > > >> > > > > [5]: > https://github.com/apache/arrow/pull/11507 > >> > > > > > > > > > > > > >> > > > > [6]: > https://github.com/apache/arrow/pull/10906 > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > Thanks, > >> > > > > > > > > > > > > >> > > > > David > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > -- > >> > > > > > > > > > > > Thanks and regards, > >> > > > > > > > > > > > Ravindra. > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > > >> > > > > > *James Duong* > >> > > > > > Lead Software Developer > >> > > > > > Bit Quill Technologies Inc. > >> > > > > > Direct: +1.604.562.6082 | jam...@bitquilltech.com > >> > > > > > https://www.bitquilltech.com > >> > > > > > > >> > > > > > This email message is for the sole use of the intended > recipient(s) and > >> > > > > may > >> > > > > > contain confidential and privileged information. Any > unauthorized > >> > > > > review, > >> > > > > > use, disclosure, or distribution is prohibited. If you are > not the > >> > > > > > intended recipient, please contact the sender by reply email > and destroy > >> > > > > > all copies of the original message. Thank you. > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > > >> > > > *James Duong* > >> > > > Lead Software Developer > >> > > > Bit Quill Technologies Inc. > >> > > > Direct: +1.604.562.6082 | jam...@bitquilltech.com > >> > > > https://www.bitquilltech.com > >> > > > > >> > > > This email message is for the sole use of the intended > recipient(s) and may > >> > > > contain confidential and privileged information. Any > unauthorized review, > >> > > > use, disclosure, or distribution is prohibited. If you are not > the > >> > > > intended recipient, please contact the sender by reply email and > destroy > >> > > > all copies of the original message. Thank you. > >> > > > > >> > > > >> > >
