Got it. Given that quite a bit of folks still use 3.1, I don’t think we would remove it unless the branch becomes inactive. Marking it as deprecated would allow us to indicate that it may not be as up-to-date and complete as other versions and some performance enhancements or even minor bug fixes may not be there. That would solve one of the concerns I raised earlier. We should discourage users from onboarding new use cases on 3.1.
I believe our doc summarizes the message pretty well. Deprecated: an engine version is no longer actively maintained. People who are still interested in the version can backport any necessary feature or bug fix from newer versions, but the community will not spend effort in achieving feature parity. Iceberg recommends users to move towards a newer version. Contributions to a deprecated version is expected to diminish over time, so that eventually no change is added to a deprecated version. https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status <https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status> Let me know if that seems like a good compromise. - Anton > On Apr 25, 2023, at 8:01 PM, Walaa Eldin Moustafa <wa.moust...@gmail.com> > wrote: > > To elaborate on LinkedIn's use case: > > * LinkedIn maintains its own fork, but we would like to keep it as close to > upstream as possible. > * +1 to Manu on migrations in large companies could take well beyond 18 > months, and it is unlikely to migrate/upgrade more frequently. > * One important use case for the Spark 3.1 module is not necessarily fixing > issues in the module itself, but fixing issues in other core modules, and > having a release that contains core fixes as well as Spark 3.1. > * That said, in the last 6 months, there have been 29 commits to Spark 3.1 > module, 50 commits to Spark 3.2 module, and 90 to Spark 3.3, in the Iceberg > master branch. It seems that Spark 3.1 is reasonably active. > > What does marking as deprecated entail in terms of deleting the code? Would > the guideline be to use 3.2 or 3.3 as an alternative? > > Thanks, > Walaa. > > > > On Tue, Apr 25, 2023 at 12:08 PM Anton Okolnychyi > <aokolnyc...@apple.com.invalid> wrote: > Ok, seems like we are in agreement to deprecate 3.1. I’ll fire a PR shortly. > > Does anyone want to go through changes in 3.3 and 3.2 and find what we missed > to cherry-pick so that we have that list in one place (e.g. create an issue)? > > Any thoughts on how to mark changes as candidates for cherry-picking? > Creating an issue? > > - Anton > >> On Apr 24, 2023, at 10:01 AM, Edgar Rodriguez >> <edgar.rodrig...@airbnb.com.INVALID >> <mailto:edgar.rodrig...@airbnb.com.INVALID>> wrote: >> >> Hi all, >> >> Thanks for the discussion. Similarly to Manu, we're in Spark 3.1.1 and >> Iceberg 1.1.0 - we backport Spark 3.1.1 fixes internally as well. It's a bit >> more complicated to move fast on Spark versions internally, mainly due to >> the number of scala customers that we have. >> >> I understand maintaining yet another Spark version is burdensome so I'm +1 >> on marking 3.1 deprecated, and I'd be happy to contribute on backports if >> needed on a community maintained branch, we'd just need to tag changes that >> may need a backport. >> >> Cheers, >> >> On Sun, Apr 23, 2023 at 4:40 PM Ryan Blue <b...@tabular.io >> <mailto:b...@tabular.io>> wrote: >> Thank you for stepping up and offering to help, Manu. I'm glad that you're >> willing to help with backports. >> >> On Sun, Apr 23, 2023 at 2:05 AM Manu Zhang <owenzhang1...@gmail.com >> <mailto:owenzhang1...@gmail.com>> wrote: >> You would just end up backporting twice. >> >> That's why I said a community maintained branch benefits us, saving one >> backport. Note the first backport is more difficult, sometimes requiring >> rewriting the PR since there would be API differences between Spark >> versions. >> The second backport will be much easier if we focus on bug fixes. >> Meanwhile, it's also easier for us to upgrade to Iceberg 1.2+ if 3.1 support >> is still available although deprecated. >> >> >> >> -- >> Ryan Blue >> Tabular >> >> >> -- >> Edgar R >
