Hi, Thanks for bringing this up.
AFAIK, this has always been an issue because we do not require all implementations to support a type prior to releasing that new type in the spec. Concrete example: map has been released for a while, but Rust does not support it yet. Same for interval, where we still do not test in integration. My understanding is that once we accept that we can bump the spec without having all implementations implement it, people will need to consult *some table*. A related issue is the *some table* bit: someone using pyarrow==2 can't tell whether they can send type X to Rust v2 via IPC. If they go to the latest table [1], they can only see the latest status (4.0 in this case). AFAIK they would need to download the source, go to the table in .rst for that release, and check whether type X is supported by Rust on that release. A way to alleviate this is to change our status table from "pass / not pass" to a "since version X.Y". This would offer the transparency into which version it was implemented since, and would also align well with the idea that Rust X.Y.Z is bundled as part of the arrow release. This does not address backward incompatible changes of the format, which is a whole different beast (e.g. do we require all implementations to change prior to releasing the new version of the format?) [1] https://arrow.apache.org/docs/status.html On Thu, Apr 29, 2021 at 2:27 AM Weston Pace <weston.p...@gmail.com> wrote: > We now have independent releases. There has been some discussion (not > sure if it was formalized) around aligning major release versions > across the languages. > > There is also a potential format change coming up (new interval type). > > I think this brings up a few questions... > > Can an arrow library adopt a new format version in a minor release? > Semantic versioning would imply yes but then I think we could get in a > situation where Arrow-C++ X.Y.Z Might not be able to communicate via > flight using Arrow-C++ X.A.B. > > In addition, is there value in aligning format adoption across languages? > > For example, if Rust adopts format version 1.1 in version 5 and > pyarrow does not then users will need to consult a table to figure out > which versions are interoperable. > > -Weston >
