Ah brilliant! Yeah, Websockets (or anything that's a basic transport and
doesn't require a language-specific SDK) would be fantastic.
In my case, streaming wouldn't be a requirement, at least not for some time
(more of a nice-to-have).
It'd be mostly OLTP-style workloads, with small response sizes (10-1,000kB).
By the way -- wanted to thank yourself and the others from the mailing list
for all the help.
Last night I was able to get a basic FlightSQL server implementation
working based on the feedback I'd got here.
Now the only challenge is not being familiar with the Arrow format +
APIs/working with vector-based data
Majority of the time was in trying to figure out how to translate JVM
arrays/objects into Arrow values.
The one thing I did have to do is override dependencies due to a problem in
netty/grpc with an
incompatible constructor signature for "PooledByteBufAllocator"
// workaround for bug with PooledByteBufAllocator
implementation("io.grpc", "grpc-netty").version {
strictly("1.44.1")
}
implementation("io.netty", "netty-all").version {
strictly("4.1.74.Final")
}
implementation("io.netty", "netty-codec").version {
strictly("4.1.74.Final")
}
On Mon, Mar 7, 2022 at 9:39 AM David Li <lidav...@apache.org> wrote:
> No worries about questions, it's always good to see how people are using
> Arrow.
>
> For tunneling Flight/gRPC over HTTP: this has been a long-standing
> question. I believe some people have had success with one of the various
> gRPC-HTTP proxies. In particular, I recall Deephaven has done this
> successfully (with some workaround for the lack of streaming methods). If
> Nate is around, maybe he can describe what they've done.
>
> There's also an ongoing effort to enable alternative transports in Flight
> [1], which would let us implement (say) a native WebSocket transport.
>
> For these methods specifically: they basically wrap Protobuf
> SerializeToString/ParseFromString so you could use them to try to implement
> your own protocol using HTTP, yes.
>
> [1]: https://github.com/apache/arrow/pull/12465
>
> -David
>
> On Mon, Mar 7, 2022, at 09:24, Gavin Ray wrote:
> > Due to the current implementation status of FlightSQL (C++/Rust/JVM only)
> >
> > I am trying to see whether it's possible to allow FlightSQL over
> something
> > like HTTP/REST so that arbitrary languages can be used.
> >
> > In the codebase, I saw these (and their deserialize counterparts):
> >
> > /// \brief Get the wire-format representation of this type.
> > /// Useful when interoperating with non-Flight systems (e.g. REST
> > /// services) that may want to return Flight types.
> > arrow::Result<std::string> SerializeToString() const;
> >
> > /**
> > * Get the serialized form of this protocol message.
> > * <p>Intended to help interoperability by allowing non-Flight services
> > to still return Flight types.
> > */
> > public ByteBuffer serialize() {
> > return ByteBuffer.wrap(toProtocol().toByteArray());
> > }
> >
> > I know this is probably very low-priority at the moment, but just wanted
> to
> > ask about whether it's even possible.
> > Thank you, and sorry for spamming the mailing list with so many questions
> > lately =)
>
