See apache/arrow-experiments [1] where there are many examples in multiple languages of doing just that, and more complicated examples of things like mixing Arrow and JSON data in a stream.
(1): Yes, that's the intent of Arrow IPC! (2): I don't think there's anything glaring off the top of my head, except for maybe the usual points to note for using Arrow in general. (I wouldn't exchange 1 row at a time, for instance, that will have a lot of overhead.) (3): The Arrow _format_ is standardized and versioned separately from Arrow _implementations_ so in general what you describe should work unless you are using something only available in a new format version (which happens rarely). You can read more at [2]. [1]: https://github.com/apache/arrow-experiments/tree/main/http [2]: https://arrow.apache.org/docs/format/Versioning.html On Wed, Jan 22, 2025, at 04:57, Amarinder Randhawa wrote: > Hello Arrow developers > > I’m exploring the use of RecordBatchStreamWriter and RecordBatchStreamReader > to serialize/deserialize Arrow data, over HTTP, as binary data payload. My > assumption is that these methods provide a machine-independent > serialization mechanism, allowing the Arrow format over HTTP to serve as a > reliable and efficient protocol for data exchange between client and server. > > Few questions: > > 1. Can I depend on these interfaces to leverage Arrow format as binary > exchange mechanism over HTTP? > > 2. Are there any limitations or considerations I should be aware of when > adopting this approach for such use cases? > > 3. Is there a standardization in place for compatibility as clients > machines may update arrow library independently (example client using arrow > version 19 for StreamReader while the server is using version 18 for > StreamWriter). > > Your insights and advice would be greatly appreciated. > > > Thanks > > Amarinder S Randhawa
