So "Flight" and "Flight SQL" are distinct projects. Flight defines RPC methods, and "Flight SQL" defines higher-level methods on top of the Flight methods. The optimization proposed is for Flight. Once/if that gets accepted and implemented, Flight SQL servers could then use it to optimize GetCatalogs: they would return a FlightInfo that has the data embedded. So yes, all the methods should get support for this once things get worked out.
On Mon, Mar 7, 2022, at 10:57, Gavin Ray wrote: > Sure, will use that JIRA issue for whatever thoughts/feedback =) > > On that note, filed the above bug here: > https://issues.apache.org/jira/browse/ARROW-15861 > > About the "two-step" thing, I guess what I mean is code like this > where you make the initial op, then get the stream: > > val catalogs: FlightInfo = client.getCatalogs() > val stream: FlightStream = > client.getStream(catalogs.endpoints[0].ticket) > while (stream.next()) { > stream.root.use { root -> println(root.contentToTSVString()) } > } > > You override two methods, "getFlightInfoCatalogs" and then > "getStreamCatalogs" > Maybe I misunderstood -- can you just return the data directly from IE > "getFlightInfoCatalogs" > > Ideally I'd love to be able to do something like: > > val catalogs = client.getCatalogs() > for (catalog in catalogs.rows) {} > > But maybe this is not really feasible/practical with how Arrow works as a > format or the architecture of Flight > > And RE: the JS implementation, TypeScript is my primary language so I'd > love to be useful there if I could =) > It's also much faster to prototype stuff in JS/TS due to lack of > compilation. > > On Mon, Mar 7, 2022 at 10:46 AM David Li <lidav...@apache.org> wrote: > >> (responses inline) >> >> On Mon, Mar 7, 2022, at 10:37, Gavin Ray wrote: >> >> >> >> Another contributor is currently working on some Java >> >> tutorials/documentation so any feedback would be helpful. >> > >> > >> > Ah, yeah this would be incredibly useful. Will compile some thoughts, >> where >> > should I share them? >> > Didn't know about the Cookbook, definitely going to be tonight's reading! >> > >> >> Would you mind putting them on the overall Jira? >> https://issues.apache.org/jira/browse/ARROW-15156 >> >> If there's questions about the cookbook, or tasks where it's not clear how >> to accomplish them, you can file issues directly on the cookbook repo too. >> >> > >> > Ah, I suppose having the small-value optimization would mostly cover your >> >> needs then? And then grpc-web or a similar bridge should suffice for >> you. >> > >> > >> > Yeah 100% >> > Wanted to ask a question on this -- is there a possibility to add the >> > "one-shot" single message RPC s for all operations? >> > >> > In my case it's mostly extra-overhead to send the first ticket, get a >> > statement handle, and then make a second call which streams the results >> > Would be awesome to have the ability to opt-in to one-shot messages for >> > both Metadata and Query operations >> >> Hmm, which other operations are you looking at? For instance, GetSchema >> takes a FlightDescriptor directly. It's really just DoGet that has that >> two-step structure. >> >> > >> > If you have details about the dependency issue, do you mind filing a Jira >> >> issue? >> >> Seems something might have changed and we should be prepared to fix it. >> >> (Flight/Java does a lot of poking at internal APIs to try to avoid >> copies.) >> > >> > >> > Absolutely, no problem. I'll revert my dep override and file an issue >> with >> > the stacktrace. >> >> Thanks! >> >> > --- >> > On a side note, I've started work on a Node.js implementation of Flight + >> > FlightSQL in the Arrow repo. >> > Never worked with gRPC but hopefully I can get the majority of the work >> > finished and file a draft PR =) >> >> That will be interesting to see. I believe the Arrow JS implementation >> could use some more attention in general. >> >> > >> > https://gist.github.com/GavinRay97/876c8e8476b18c8eb01cb6e8f807bf28 >> > >> > On Mon, Mar 7, 2022 at 9:55 AM David Li <lidav...@apache.org> wrote: >> > >> >> Cool - if you have API questions, feel free to send them here or >> >> u...@arrow.apache.org. Another contributor is currently working on some >> >> Java tutorials/documentation so any feedback would be helpful. There's >> also >> >> some basic recipes here: https://github.com/apache/arrow-cookbook/ >> >> >> >> Ah, I suppose having the small-value optimization would mostly cover >> your >> >> needs then? And then grpc-web or a similar bridge should suffice for >> you. >> >> >> >> If you have details about the dependency issue, do you mind filing a >> Jira >> >> issue? Seems something might have changed and we should be prepared to >> fix >> >> it. (Flight/Java does a lot of poking at internal APIs to try to avoid >> >> copies.) >> >> >> >> Thanks, >> >> David >> >> >> >> On Mon, Mar 7, 2022, at 09:48, Gavin Ray wrote: >> >> > Ah brilliant! Yeah, Websockets (or anything that's a basic transport >> and >> >> > doesn't require a language-specific SDK) would be fantastic. >> >> > >> >> > In my case, streaming wouldn't be a requirement, at least not for some >> >> time >> >> > (more of a nice-to-have). >> >> > It'd be mostly OLTP-style workloads, with small response sizes >> >> (10-1,000kB). >> >> > >> >> > By the way -- wanted to thank yourself and the others from the mailing >> >> list >> >> > for all the help. >> >> > Last night I was able to get a basic FlightSQL server implementation >> >> > working based on the feedback I'd got here. >> >> > >> >> > Now the only challenge is not being familiar with the Arrow format + >> >> > APIs/working with vector-based data >> >> > Majority of the time was in trying to figure out how to translate JVM >> >> > arrays/objects into Arrow values. >> >> > >> >> > The one thing I did have to do is override dependencies due to a >> problem >> >> in >> >> > netty/grpc with an >> >> > incompatible constructor signature for "PooledByteBufAllocator" >> >> > >> >> > // workaround for bug with PooledByteBufAllocator >> >> > implementation("io.grpc", "grpc-netty").version { >> >> > strictly("1.44.1") >> >> > } >> >> > implementation("io.netty", "netty-all").version { >> >> > strictly("4.1.74.Final") >> >> > } >> >> > implementation("io.netty", "netty-codec").version { >> >> > strictly("4.1.74.Final") >> >> > } >> >> > >> >> > On Mon, Mar 7, 2022 at 9:39 AM David Li <lidav...@apache.org> wrote: >> >> > >> >> >> No worries about questions, it's always good to see how people are >> using >> >> >> Arrow. >> >> >> >> >> >> For tunneling Flight/gRPC over HTTP: this has been a long-standing >> >> >> question. I believe some people have had success with one of the >> various >> >> >> gRPC-HTTP proxies. In particular, I recall Deephaven has done this >> >> >> successfully (with some workaround for the lack of streaming >> methods). >> >> If >> >> >> Nate is around, maybe he can describe what they've done. >> >> >> >> >> >> There's also an ongoing effort to enable alternative transports in >> >> Flight >> >> >> [1], which would let us implement (say) a native WebSocket transport. >> >> >> >> >> >> For these methods specifically: they basically wrap Protobuf >> >> >> SerializeToString/ParseFromString so you could use them to try to >> >> implement >> >> >> your own protocol using HTTP, yes. >> >> >> >> >> >> [1]: https://github.com/apache/arrow/pull/12465 >> >> >> >> >> >> -David >> >> >> >> >> >> On Mon, Mar 7, 2022, at 09:24, Gavin Ray wrote: >> >> >> > Due to the current implementation status of FlightSQL (C++/Rust/JVM >> >> only) >> >> >> > >> >> >> > I am trying to see whether it's possible to allow FlightSQL over >> >> >> something >> >> >> > like HTTP/REST so that arbitrary languages can be used. >> >> >> > >> >> >> > In the codebase, I saw these (and their deserialize counterparts): >> >> >> > >> >> >> > /// \brief Get the wire-format representation of this type. >> >> >> > /// Useful when interoperating with non-Flight systems (e.g. REST >> >> >> > /// services) that may want to return Flight types. >> >> >> > arrow::Result<std::string> SerializeToString() const; >> >> >> > >> >> >> > /** >> >> >> > * Get the serialized form of this protocol message. >> >> >> > * <p>Intended to help interoperability by allowing non-Flight >> >> services >> >> >> > to still return Flight types. >> >> >> > */ >> >> >> > public ByteBuffer serialize() { >> >> >> > return ByteBuffer.wrap(toProtocol().toByteArray()); >> >> >> > } >> >> >> > >> >> >> > I know this is probably very low-priority at the moment, but just >> >> wanted >> >> >> to >> >> >> > ask about whether it's even possible. >> >> >> > Thank you, and sorry for spamming the mailing list with so many >> >> questions >> >> >> > lately =) >> >> >> >> >> >>
