Thank you Aldrin and David for sharing the helpful links. This really unblocked me and the https://github.com/apache/arrow-experiments/blob/main/http was very directed to speed up the use of Arrow IPC.
David - Although it is mentioned that the Arrow Format major version has not changed till now (how will the upgrade work whenever this change happens). Much appreciated. - Amarinder S Randhawa On Tue, 21 Jan 2025 at 15:58, Aldrin <octalene....@pm.me> wrote: > Hello! > > Can I depend on these interfaces to leverage Arrow format as binary > exchange mechanism over HTTP? > > Yes. You can see [1] for a bit of discussion and some github links. But, > the short answer is that the stream writer and stream reader interfaces are > convenient interfaces to data movement over IPC streams, and is good over > any medium (AFAIK). > > Is there a standardization in place for compatibility... > > Yes! It is the arrow columnar format [2] which is versioned separately > from any particular implementation of it. For example, the versions you're > talking about are for the C++ implementation. > > As for your other question, that's a bit broad and maybe can be mostly > addressed via a new blog post [3] or some future post in that series. Plus, > there's a variety of information you can find by looking around. If you > have a more specific question feel free to follow up on the mailing list. > > > > [1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191 > > [2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar > > [3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/ > > > Sent from Proton Mail <https://proton.me/mail/home> for iOS > > > On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randh...@gmail.com > <On+Tue,+Jan+21,+2025+at+11:57,+Amarinder+Randhawa+%3C%3Ca+href=>> wrote: > > Hello Arrow developers > > I’m exploring the use of RecordBatchStreamWriter and > RecordBatchStreamReader > to serialize/deserialize Arrow data, over HTTP, as binary data payload. My > assumption is that these methods provide a machine-independent > serialization mechanism, allowing the Arrow format over HTTP to serve as a > reliable and efficient protocol for data exchange between client and > server. > > Few questions: > > 1. Can I depend on these interfaces to leverage Arrow format as binary > exchange mechanism over HTTP? > > 2. Are there any limitations or considerations I should be aware of when > adopting this approach for such use cases? > > 3. Is there a standardization in place for compatibility as clients > machines may update arrow library independently (example client using > arrow > version 19 for StreamReader while the server is using version 18 for > StreamWriter). > > Your insights and advice would be greatly appreciated. > > > Thanks > > Amarinder S Randhawa > >
