Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-18 Thread Ian Cook
Thanks to everyone who has contributed to this work so far. We now have simple HTTP client/server examples in 10 languages, all tested and verified to interoperate: https://github.com/apache/arrow-experiments/tree/main/http/get_simple There is an umbrella issue tracking the next planned tasks: htt

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-11 Thread Bryce Mecum
I'd be happy to contribute C# and Ruby examples. I'll work on those this week. On Tue, Mar 5, 2024 at 7:03 PM Ian Cook wrote: > > Update on recent progress in this Arrow-over-HTTP project: > > I cleaned up the minimal examples of HTTP clients and servers and > moved them into a directory in the A

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-11 Thread Andrew Lamb
Update -- turns out there was already a Rust client/server -- linked to the ticket now On Mon, Mar 11, 2024 at 3:07 PM Andrew Lamb wrote: > I sadly don't have time to help with this directly, however, I did file a > ticket with the request to help with a Rust prototype [1]. Hopefully we'll > get

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-11 Thread Andrew Lamb
I sadly don't have time to help with this directly, however, I did file a ticket with the request to help with a Rust prototype [1]. Hopefully we'll get a taker [1] https://github.com/apache/arrow-rs/issues/5496 On Tue, Mar 5, 2024 at 11:03 PM Ian Cook wrote: > Update on recent progress in this

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-03-05 Thread Ian Cook
Update on recent progress in this Arrow-over-HTTP project: I cleaned up the minimal examples of HTTP clients and servers and moved them into a directory in the Arrow Experiments repo: https://github.com/apache/arrow-experiments/tree/main/http So far there are client examples in six languages and

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-02-02 Thread Alessandro Molina
On Wed, Dec 6, 2023 at 7:45 PM Ian Cook wrote: > > I am interested to hear more perspectives on this. My perspective is > that we should recommend using HTTP conventions to keep clean > separation between the Arrow-formatted binary data payloads and the > various application-specific fields. This

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2024-01-08 Thread Wes McKinney
hi all — I was just catching up on e-mail threads and wanted to give a few historical comments on this. When we were assembling the Arrow PMC and committing to do the project in 2015, standardizing Arrow-over-REST was always something that was on the TODO list — at that time we didn't have the IPC

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2023-12-06 Thread Ian Cook
I just remembered that there is an unused "Arrow Experiments" repo [1] which Wes created a few years ago [2]. That seems like a more appropriate place to open PRs like this one. If there are no objections, I will start using that repo for these Arrow-over-HTTP PRs. [1] https://github.com/apache/ar

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2023-12-06 Thread Ian Cook
Antoine, Thank you for taking a look. I agree—these are basic examples intended to prove the concept and answer fundamental questions. Next I intend to expand the set of examples to cover more complex cases. > This might necessitate some kind of framing layer, or a > standardized delimiter. I am

Re: [DISCUSS] Conventions for transporting Arrow data over HTTP

2023-12-06 Thread Antoine Pitrou
Hi, While this looks like a nice start, I would expect more precise recommendations for writing non-trivial services. Especially, one question is how to send both an application-specific POST request and an Arrow stream, or an application-specific GET response and an Arrow stream. This migh