> David - Although it is mentioned that the Arrow Format major version has > not changed till now (how will the upgrade work whenever this change > happens).
A *major* version change is extremely unlikely at this point and I don't think we can say how it would work. It would almost certainly be backwards-incompatible (else it wouldn't be a major version change), but I don't think I'd lose sleep planning for it coming anytime soon. On Sat, Feb 1, 2025, at 03:03, Amarinder Randhawa wrote: > Thank you Aldrin and David for sharing the helpful links. > This really unblocked me and the > https://github.com/apache/arrow-experiments/blob/main/http was very > directed to speed up the use of Arrow IPC. > > David - Although it is mentioned that the Arrow Format major version has > not changed till now (how will the upgrade work whenever this change > happens). > > Much appreciated. > - Amarinder S Randhawa > > On Tue, 21 Jan 2025 at 15:58, Aldrin <octalene....@pm.me> wrote: > >> Hello! >> >> Can I depend on these interfaces to leverage Arrow format as binary >> exchange mechanism over HTTP? >> >> Yes. You can see [1] for a bit of discussion and some github links. But, >> the short answer is that the stream writer and stream reader interfaces are >> convenient interfaces to data movement over IPC streams, and is good over >> any medium (AFAIK). >> >> Is there a standardization in place for compatibility... >> >> Yes! It is the arrow columnar format [2] which is versioned separately >> from any particular implementation of it. For example, the versions you're >> talking about are for the C++ implementation. >> >> As for your other question, that's a bit broad and maybe can be mostly >> addressed via a new blog post [3] or some future post in that series. Plus, >> there's a variety of information you can find by looking around. If you >> have a more specific question feel free to follow up on the mailing list. >> >> >> >> [1]: https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191 >> >> [2]: https://arrow.apache.org/docs/format/Columnar.html#format-columnar >> >> [3]: https://arrow.apache.org/blog/2025/01/10/arrow-result-transfer/ >> >> >> Sent from Proton Mail <https://proton.me/mail/home> for iOS >> >> >> On Tue, Jan 21, 2025 at 11:57, Amarinder Randhawa <asr.randh...@gmail.com >> <On+Tue,+Jan+21,+2025+at+11:57,+Amarinder+Randhawa+%3C%3Ca+href=>> wrote: >> >> Hello Arrow developers >> >> I’m exploring the use of RecordBatchStreamWriter and >> RecordBatchStreamReader >> to serialize/deserialize Arrow data, over HTTP, as binary data payload. My >> assumption is that these methods provide a machine-independent >> serialization mechanism, allowing the Arrow format over HTTP to serve as a >> reliable and efficient protocol for data exchange between client and >> server. >> >> Few questions: >> >> 1. Can I depend on these interfaces to leverage Arrow format as binary >> exchange mechanism over HTTP? >> >> 2. Are there any limitations or considerations I should be aware of when >> adopting this approach for such use cases? >> >> 3. Is there a standardization in place for compatibility as clients >> machines may update arrow library independently (example client using >> arrow >> version 19 for StreamReader while the server is using version 18 for >> StreamWriter). >> >> Your insights and advice would be greatly appreciated. >> >> >> Thanks >> >> Amarinder S Randhawa >> >>
