Hi, The major problem that we are addressing with an independent release cycle is that the large majority of our users do not use released versions, neither from ASF archives nor from Cargo crates. They use a git hash commit. This is a problem because our git hashes are *de facto* releases.
This hints that our development is not stable and we should increase the cadency over which we release, even if backward incompatibly. Doing so mitigates the risks associated with unsigned and unvoted software that our processes are aimed to address. Assuming SemVer, a major version every X months forbids this, as it freezes the cadency in both directions (faster or slower). I do agree that a release that bundles all implementations and ships them as "this bundle is guaranteed to be spec-interoperable" is very valuable, should be preserved, and Rust should be part of it. It offers strong guarantees of interoperability across implementations at a specific "code-point" across implementations. To address these two requirements, my suggestion is that we continue to release arrow/apache as is, but instead of containing every implementation versioned as X, it contains every implementation with the latest released version of their API *and* spec-compliant with the latest apache/arrow (as per integration tests). I.e. the "code-point" for Rust is a specific version, which maps to a uniquely identifiable source code. Note that this helps in both directions: it would also make it easier for e.g. pyarrow to be major-released less often, if their maintainers believe that its X.0.0 API can remain stable for some more time. > one rationale from our current policy of "everyone releases together" is that you don't > have to guess as much whether (for example) Arrow Java 3.0 and Arrow Rust 3.0 > are compatible and using the same format. I agree that we would lose this. I would document which version the release refers to. E.g. "this release contains the following releases, published in their respective package managers: Java 3.0, pyarrow 3.0, Rust 4.2.1, Julia 2.2.1, and also available at [link to archive]". This allows people to point to specific versions if they want this assurance in e.g. a cross-language application, or in a contract. Best, Jorge On Sat, Apr 10, 2021 at 4:27 AM QP Hou <q...@scribd.com.invalid> wrote: > On Fri, Apr 9, 2021 at 4:57 PM Weston Pace <weston.p...@gmail.com> wrote: > > Note, these problems technically exist now with the concept that any > > language can release a patch at any time. Also, since Rust isn't > > directly compiling against other Arrow libs and we are only talking > > about interoperability it's probably not going to be too big of a > > deal. Still, worth giving some thought ahead of time. > > It looks like the requirements we have here are: > > * Catch integration issues between development trunks between two > repos as soon as we can > * Avoid development trunk of a repo from blocking releases of another repo > > How about we always run the integration tests against the latest > releases in both repos by default. On top of that, we run integration > tests in the Rust repo against nightly Arrow to help catch integration > issues between development trunks. In the Rust repo, we could enforce > the rule that a Rust release will only be cut if the release candidate > passes the integration tests for both the latest released and nightly > Arrow code base. > > This way, we will still be able to catch integration issues between > development code trunks on a daily basis. It would be Rust developers' > responsibility to report these integration errors back to the main > repo. > > From the main Arrow repo's point of view, there is less work since it > only needs to care about the last stable Rust release. > > From the Rust repo's point of view, we are effectively trading extra > maintenance burden for more release flexibility. > > Thanks, > QP >
