(responses inline) On Mon, Mar 7, 2022, at 10:37, Gavin Ray wrote: >> >> Another contributor is currently working on some Java >> tutorials/documentation so any feedback would be helpful. > > > Ah, yeah this would be incredibly useful. Will compile some thoughts, where > should I share them? > Didn't know about the Cookbook, definitely going to be tonight's reading! >
Would you mind putting them on the overall Jira? https://issues.apache.org/jira/browse/ARROW-15156 If there's questions about the cookbook, or tasks where it's not clear how to accomplish them, you can file issues directly on the cookbook repo too. > > Ah, I suppose having the small-value optimization would mostly cover your >> needs then? And then grpc-web or a similar bridge should suffice for you. > > > Yeah 100% > Wanted to ask a question on this -- is there a possibility to add the > "one-shot" single message RPC s for all operations? > > In my case it's mostly extra-overhead to send the first ticket, get a > statement handle, and then make a second call which streams the results > Would be awesome to have the ability to opt-in to one-shot messages for > both Metadata and Query operations Hmm, which other operations are you looking at? For instance, GetSchema takes a FlightDescriptor directly. It's really just DoGet that has that two-step structure. > > If you have details about the dependency issue, do you mind filing a Jira >> issue? >> Seems something might have changed and we should be prepared to fix it. >> (Flight/Java does a lot of poking at internal APIs to try to avoid copies.) > > > Absolutely, no problem. I'll revert my dep override and file an issue with > the stacktrace. Thanks! > --- > On a side note, I've started work on a Node.js implementation of Flight + > FlightSQL in the Arrow repo. > Never worked with gRPC but hopefully I can get the majority of the work > finished and file a draft PR =) That will be interesting to see. I believe the Arrow JS implementation could use some more attention in general. > > https://gist.github.com/GavinRay97/876c8e8476b18c8eb01cb6e8f807bf28 > > On Mon, Mar 7, 2022 at 9:55 AM David Li <lidav...@apache.org> wrote: > >> Cool - if you have API questions, feel free to send them here or >> u...@arrow.apache.org. Another contributor is currently working on some >> Java tutorials/documentation so any feedback would be helpful. There's also >> some basic recipes here: https://github.com/apache/arrow-cookbook/ >> >> Ah, I suppose having the small-value optimization would mostly cover your >> needs then? And then grpc-web or a similar bridge should suffice for you. >> >> If you have details about the dependency issue, do you mind filing a Jira >> issue? Seems something might have changed and we should be prepared to fix >> it. (Flight/Java does a lot of poking at internal APIs to try to avoid >> copies.) >> >> Thanks, >> David >> >> On Mon, Mar 7, 2022, at 09:48, Gavin Ray wrote: >> > Ah brilliant! Yeah, Websockets (or anything that's a basic transport and >> > doesn't require a language-specific SDK) would be fantastic. >> > >> > In my case, streaming wouldn't be a requirement, at least not for some >> time >> > (more of a nice-to-have). >> > It'd be mostly OLTP-style workloads, with small response sizes >> (10-1,000kB). >> > >> > By the way -- wanted to thank yourself and the others from the mailing >> list >> > for all the help. >> > Last night I was able to get a basic FlightSQL server implementation >> > working based on the feedback I'd got here. >> > >> > Now the only challenge is not being familiar with the Arrow format + >> > APIs/working with vector-based data >> > Majority of the time was in trying to figure out how to translate JVM >> > arrays/objects into Arrow values. >> > >> > The one thing I did have to do is override dependencies due to a problem >> in >> > netty/grpc with an >> > incompatible constructor signature for "PooledByteBufAllocator" >> > >> > // workaround for bug with PooledByteBufAllocator >> > implementation("io.grpc", "grpc-netty").version { >> > strictly("1.44.1") >> > } >> > implementation("io.netty", "netty-all").version { >> > strictly("4.1.74.Final") >> > } >> > implementation("io.netty", "netty-codec").version { >> > strictly("4.1.74.Final") >> > } >> > >> > On Mon, Mar 7, 2022 at 9:39 AM David Li <lidav...@apache.org> wrote: >> > >> >> No worries about questions, it's always good to see how people are using >> >> Arrow. >> >> >> >> For tunneling Flight/gRPC over HTTP: this has been a long-standing >> >> question. I believe some people have had success with one of the various >> >> gRPC-HTTP proxies. In particular, I recall Deephaven has done this >> >> successfully (with some workaround for the lack of streaming methods). >> If >> >> Nate is around, maybe he can describe what they've done. >> >> >> >> There's also an ongoing effort to enable alternative transports in >> Flight >> >> [1], which would let us implement (say) a native WebSocket transport. >> >> >> >> For these methods specifically: they basically wrap Protobuf >> >> SerializeToString/ParseFromString so you could use them to try to >> implement >> >> your own protocol using HTTP, yes. >> >> >> >> [1]: https://github.com/apache/arrow/pull/12465 >> >> >> >> -David >> >> >> >> On Mon, Mar 7, 2022, at 09:24, Gavin Ray wrote: >> >> > Due to the current implementation status of FlightSQL (C++/Rust/JVM >> only) >> >> > >> >> > I am trying to see whether it's possible to allow FlightSQL over >> >> something >> >> > like HTTP/REST so that arbitrary languages can be used. >> >> > >> >> > In the codebase, I saw these (and their deserialize counterparts): >> >> > >> >> > /// \brief Get the wire-format representation of this type. >> >> > /// Useful when interoperating with non-Flight systems (e.g. REST >> >> > /// services) that may want to return Flight types. >> >> > arrow::Result<std::string> SerializeToString() const; >> >> > >> >> > /** >> >> > * Get the serialized form of this protocol message. >> >> > * <p>Intended to help interoperability by allowing non-Flight >> services >> >> > to still return Flight types. >> >> > */ >> >> > public ByteBuffer serialize() { >> >> > return ByteBuffer.wrap(toProtocol().toByteArray()); >> >> > } >> >> > >> >> > I know this is probably very low-priority at the moment, but just >> wanted >> >> to >> >> > ask about whether it's even possible. >> >> > Thank you, and sorry for spamming the mailing list with so many >> questions >> >> > lately =) >> >> >>
