Sorry for the late reply here! These look reasonable to me. I think that this will help us reason about trade-offs next time we have a release issue like the current one. We should simply mark the 3.2 support as beta and get the release out next time. I also think that we should not create situations with regressions like this again. We should probably have copied the old MERGE/UPDATE/DELETE plans into 3.2 to avoid a regression and then update them to the new implementations later, without affecting releases.
Thanks for writing this up, Jack! Ryan On Wed, Dec 15, 2021 at 7:18 PM Jack Ye <yezhao...@gmail.com> wrote: > Hi everyone, > > As a part of the ongoing 0.13.0 release, we are starting to formally > support multiple engine versions for Spark, Flink and Hive. I think it is > worth defining a formal process for us to add a new supported version, > maintain existing versions and deprecate old versions. We briefly touched > this topic when doing the refactoring, but I think now is a good time to > formalize it and place it as a part of the Iceberg public documentation. As > a starter for brainstorming, here is the process I think: > > Each engine has the following lifecycle states: > > 1. *Beta*: an engine supported is added, but still in the experimental > stage. Maybe the engine version itself is still in preview (e.g. Spark > 3.0.0-preview), or the engine does not yet have full feature > compatibility compared to old versions yet. This state allows us to > release an engine version support without the need to wait for feature > parity, shortening the release time. > > 2. *Maintained*: an engine version is being actively maintained by the > community. Users can expect feature parity for most features across all the > maintained versions. If a feature has to leverage some new engine > functionalities that older versions don't have, then feature parity is not > required. For code contributors, > - New features should always be prioritized first in the latest version > (the latest version could be a maintained or beta version) > - For features that could be backported, the contributor is encouraged to > either also perform backports in separated PRs, or at least create some > issues to track the backport. > - If the change is small enough like a few lines, updating all versions at > once is good enough. Otherwise, using separated PRs for each version is > recommended. > > 3. *Deprecating*: an engine version is no longer actively maintained. > People who are still interested in the version can backport any necessary > feature or bug fix from newer versions, but the community will not spend > effort in achieving feature parity. We recommend users to move towards a > newer version, and we expect contributions to the specific version to > diminish over time, and eventually no change is added to the version. At > that time we can move the version to the end of life. > > 4. *End-of-life*: a vote can be initiated to fully remove a deprecating > version out of the Iceberg repo to mark as its end of life. I am not sure > if we should remove all the code, but I think it would help push people > forward and keep the repository healthy. > > With the lifecycle states described above, we will add 1 doc section under > each engine to describe the current engine version support status. A PR > will be needed to perform any state transition, and that could serve as the > place to discuss if the transition is appropriate or not. > > Any thoughts about the process? > > Best, > Jack Ye > -- Ryan Blue Tabular
