Hi Adam, Arrow Flight can be used to provide an RPC framework that returns datasets (sent over the wire as arrow buffers) and exposes them from a FlightClient as Arrow RecordBatches without serialization. Is this what you mean by remote datasets? Arrow Flight SQL is an application layer built on top of Arrow Flight that standardizes remote execution of SQL queries, getting catalog information, getting SQL capabilities, and other access-related concepts. Arrow Flight SQL is intended to provide a universal user-facing front end for existing SQL-capable database engines.
Neither are really intended for computation, just remote access. On Tue, Apr 12, 2022 at 12:51 PM Adam Lippai <a...@rigo.sk> wrote: > Hi, > > I saw really nice features like groupby and join developed recently. > I like how Dataset is supported for joins and how streamed processing is > gaining momentum in Arrow. > > Does Apache Arrow have the concept of remote datasets eg using Arrow > Flight? Or will this happen directly using S3 and other protocols only? I > know some work has started in Substrait, but that might be a whole new > level of integration, hence my question focusing on data first. > > I was trying to browse the JIRA issues, but the future picture wasn't clear > based on that > > Best regards, > Adam Lippai > -- *James Duong* Lead Software Developer Bit Quill Technologies Inc. Direct: +1.604.562.6082 | jam...@bitquilltech.com https://www.bitquilltech.com This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
