Hello! Can I depend on these interfaces to leverage Arrow format as binary exchange mechanism over HTTP?Yes. You can see [1] for a bit of discussion and some github links. But, the short answer is that the stream writer and stream reader interfaces are convenient interfaces to data movement over IPC streams, and is good over any medium (AFAIK). Is there a standardization in place for compatibility...Yes! It is the arrow columnar format [2] which is versioned separately from any particular implementation of it. For example, the versions you're talking about are for the C++ implementation. As for your other question, that's a bit broad and maybe can be mostly addressed via a new blog post [3] or some future post in that series. Plus, there's a variety of information you can find by looking around. If you have a more specific question feel free to follow up on the mailing list.
[1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191 [2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar [3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/ Sent from Proton Mail for iOS On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randh...@gmail.com> wrote: Hello Arrow developers I’m exploring the use of RecordBatchStreamWriter and RecordBatchStreamReader to serialize/deserialize Arrow data, over HTTP, as binary data payload. My assumption is that these methods provide a machine-independent serialization mechanism, allowing the Arrow format over HTTP to serve as a reliable and efficient protocol for data exchange between client and server. Few questions: 1. Can I depend on these interfaces to leverage Arrow format as binary exchange mechanism over HTTP? 2. Are there any limitations or considerations I should be aware of when adopting this approach for such use cases? 3. Is there a standardization in place for compatibility as clients machines may update arrow library independently (example client using arrow version 19 for StreamReader while the server is using version 18 for StreamWriter). Your insights and advice would be greatly appreciated. Thanks Amarinder S Randhawa
signature.asc
Description: OpenPGP digital signature
