Yes, that sounds like a good compromise. Initially I was looking at deprecation guidelines in [1], but I see you are referring to [2].
[1] https://iceberg.apache.org/contribute/ [2] https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status On Wed, Apr 26, 2023 at 8:10 AM Anton Okolnychyi <aokolnyc...@apple.com.invalid> wrote: > Got it. Given that quite a bit of folks still use 3.1, I don’t think we > would remove it unless the branch becomes inactive. Marking it as > deprecated would allow us to indicate that it may not be as up-to-date and > complete as other versions and some performance enhancements or even minor > bug fixes may not be there. That would solve one of the concerns I raised > earlier. We should discourage users from onboarding new use cases on 3.1. > > I believe our doc summarizes the message pretty well. > > > 1. *Deprecated*: an engine version is no longer actively maintained. > People who are still interested in the version can backport any necessary > feature or bug fix from newer versions, but the community will not spend > effort in achieving feature parity. Iceberg recommends users to move > towards a newer version. Contributions to a deprecated version is expected > to diminish over time, so that eventually no change is added to a > deprecated version. > > > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status > > Let me know if that seems like a good compromise. > > - Anton > > > On Apr 25, 2023, at 8:01 PM, Walaa Eldin Moustafa <wa.moust...@gmail.com> > wrote: > > To elaborate on LinkedIn's use case: > > * LinkedIn maintains its own fork, but we would like to keep it as close > to upstream as possible. > * +1 to Manu on migrations in large companies could take well beyond 18 > months, and it is unlikely to migrate/upgrade more frequently. > * One important use case for the Spark 3.1 module is not necessarily > fixing issues in the module itself, but fixing issues in other core > modules, and having a release that contains core fixes as well as Spark 3.1. > * That said, in the last 6 months, there have been 29 commits to Spark 3.1 > module, 50 commits to Spark 3.2 module, and 90 to Spark 3.3, in the Iceberg > master branch. It seems that Spark 3.1 is reasonably active. > > What does marking as deprecated entail in terms of deleting the code? > Would the guideline be to use 3.2 or 3.3 as an alternative? > > Thanks, > Walaa. > > > > On Tue, Apr 25, 2023 at 12:08 PM Anton Okolnychyi < > aokolnyc...@apple.com.invalid> wrote: > >> Ok, seems like we are in agreement to deprecate 3.1. I’ll fire a PR >> shortly. >> >> Does anyone want to go through changes in 3.3 and 3.2 and find what we >> missed to cherry-pick so that we have that list in one place (e.g. create >> an issue)? >> >> Any thoughts on how to mark changes as candidates for cherry-picking? >> Creating an issue? >> >> - Anton >> >> On Apr 24, 2023, at 10:01 AM, Edgar Rodriguez < >> edgar.rodrig...@airbnb.com.INVALID> wrote: >> >> Hi all, >> >> Thanks for the discussion. Similarly to Manu, we're in Spark 3.1.1 and >> Iceberg 1.1.0 - we backport Spark 3.1.1 fixes internally as well. It's a >> bit more complicated to move fast on Spark versions internally, mainly due >> to the number of scala customers that we have. >> >> I understand maintaining yet another Spark version is burdensome so >> I'm +1 on marking 3.1 deprecated, and I'd be happy to contribute on >> backports if needed on a community maintained branch, we'd just need to tag >> changes that may need a backport. >> >> Cheers, >> >> On Sun, Apr 23, 2023 at 4:40 PM Ryan Blue <b...@tabular.io> wrote: >> >>> Thank you for stepping up and offering to help, Manu. I'm glad that >>> you're willing to help with backports. >>> >>> On Sun, Apr 23, 2023 at 2:05 AM Manu Zhang <owenzhang1...@gmail.com> >>> wrote: >>> >>>> You would just end up backporting twice. >>>> >>>> >>>> That's why I said a community maintained branch benefits us, saving one >>>> backport. Note the first backport is more difficult, sometimes requiring >>>> rewriting the PR since there would be API differences between Spark >>>> versions. >>>> The second backport will be much easier if we focus on bug fixes. >>>> Meanwhile, it's also easier for us to upgrade to Iceberg 1.2+ if 3.1 >>>> support is still available although deprecated. >>>> >>>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >> >> >> -- >> Edgar R >> >> >> >
