Re: Binary compatibility of pyarrow.serialize

2019-08-23 Thread Wes McKinney
That said, the protocol data produced now by `RecordBatchStreamWriter` should be readable in 1.0.0 and beyond. `pyarrow.serialize` is only intended for transient storage. We should add some language to the docstring for this function to explain that it is distinct from the Arrow IPC format (which h

Re: Binary compatibility of pyarrow.serialize

2019-08-23 Thread Antoine Pitrou
Hi Yevgeni, I don't think we have ever promised binary stability of the pyarrow.serialize() protocol. Binary compatibility starting from 1.0.0 is about the Arrow in-memory format and the Arrow IPC format (i.e. how Arrow arrays, tables... are laid out and how their metadata is encoded on the wir

Binary compatibility of pyarrow.serialize

2019-08-22 Thread Yevgeni Litvin
In our system we are using arrow serialization as it showed excellent deserialization speed. However, seems that we made a mistake by persisting the streams into a long-term storage as the serialized data appears to be incompatible between versions. According to the release notes of 0.14.0 it appea