Re: Independent releases and format version

2021-04-29 Thread Wes McKinney
When in doubt, return to the format versioning policy: https://arrow.apache.org/docs/format/Versioning.html The interval type addition is a MINOR version increase in the format version. As a result of the forward compatibility policy, old readers are expected to be able to recognize when they see

Re: Independent releases and format version

2021-04-29 Thread Antoine Pitrou
Le 29/04/2021 à 02:26, Weston Pace a écrit : There is also a potential format change coming up (new interval type). Ok, so more accurately, it is not a format change, it's a format addition ;-) This sounds pedantic but a format change would potentially break compatibility (for example if

Re: Independent releases and format version

2021-04-29 Thread Antoine Pitrou
Le 29/04/2021 à 02:26, Weston Pace a écrit : We now have independent releases. There has been some discussion (not sure if it was formalized) around aligning major release versions across the languages. There is also a potential format change coming up (new interval type). I think this bring

Re: Independent releases and format version

2021-04-29 Thread Weston Pace
Ok, so it sounds less like a "library" dependency and more like the relationship between C++ specification and compiler or between browser features and browsers. Does an implementation have to implement the 1.0 spec before implementing features in the 1.1 spec? I would assume no. Given that we m

Re: Independent releases and format version

2021-04-28 Thread Micah Kornfield
> > A way to alleviate this is to change our status table from "pass / not > pass" to a "since version X.Y". This would offer the transparency into > which version it was implemented since, and would also align well with the > idea that Rust X.Y.Z is bundled as part of the arrow release. Great id

Re: Independent releases and format version

2021-04-28 Thread Jorge Cardoso Leitão
Hi, Thanks for bringing this up. AFAIK, this has always been an issue because we do not require all implementations to support a type prior to releasing that new type in the spec. Concrete example: map has been released for a while, but Rust does not support it yet. Same for interval, where we st

Re: Independent releases and format version

2021-04-28 Thread Micah Kornfield
Hi Weston, I think one other issue that we might want to discuss is releasing the format separately (i.e. as soon as we make a change to it). My thoughts: Can an arrow library adopt a new format version in a minor release? I agree the answer is yes, and some issues might arise if minor releases

Independent releases and format version

2021-04-28 Thread Weston Pace
We now have independent releases. There has been some discussion (not sure if it was formalized) around aligning major release versions across the languages. There is also a potential format change coming up (new interval type). I think this brings up a few questions... Can an arrow library ado