+1 for deprecating 3.2. I'd probably have at least one release before dropping, however.
I really prefer to drop 3.1 but we can still include it in 1.4 if there is demand for it in the community. Is there anyone still expecting Spark 3.1 jars being released in 1.4? On 2023/09/20 22:44:29 Ryan Blue wrote: > I think at a minimum, we should also mark 3.2 as deprecated like we did for > 3.1, now that 3.5 is supported. > > I'd also be in favor of removing 3.1. 3.1 was released more than 30 months > ago (March 2021) and Spark's policy is that releases are only patched for > 18 months. Also, 3.2 was initially released almost 2 years ago (October > 2021) and is also past the maintenance window in the Spark community. > > I'd be up for removing both 3.1 and 3.2, but I'd be fine removing 3.1 and > keeping 3.2 in order to have a release where it is deprecated. > > On Wed, Sep 20, 2023 at 2:54 PM Anton Okolnychyi <aokolnyc...@apache.org> > wrote: > > > Just checking in to see how we feel about Spark 3.1 now. We support 5 > > different Spark versions at this point: 3.1, 3.2, 3.3, 3.4 and 3.5. Any > > thoughts on making Iceberg 1.4 the last release with Spark 3.1? > > > > On 2023/04/27 04:58:06 Walaa Eldin Moustafa wrote: > > > Yes, that sounds like a good compromise. Initially I was looking at > > > deprecation guidelines in [1], but I see you are referring to [2]. > > > > > > [1] https://iceberg.apache.org/contribute/ > > > [2] > > > > > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status > > > > > > On Wed, Apr 26, 2023 at 8:10 AM Anton Okolnychyi > > > <aokolnyc...@apple.com.invalid> wrote: > > > > > > > Got it. Given that quite a bit of folks still use 3.1, I don’t think we > > > > would remove it unless the branch becomes inactive. Marking it as > > > > deprecated would allow us to indicate that it may not be as up-to-date > > and > > > > complete as other versions and some performance enhancements or even > > minor > > > > bug fixes may not be there. That would solve one of the concerns I > > raised > > > > earlier. We should discourage users from onboarding new use cases on > > 3.1. > > > > > > > > I believe our doc summarizes the message pretty well. > > > > > > > > > > > > 1. *Deprecated*: an engine version is no longer actively maintained. > > > > People who are still interested in the version can backport any > > necessary > > > > feature or bug fix from newer versions, but the community will not > > spend > > > > effort in achieving feature parity. Iceberg recommends users to move > > > > towards a newer version. Contributions to a deprecated version is > > expected > > > > to diminish over time, so that eventually no change is added to a > > > > deprecated version. > > > > > > > > > > > > > > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status > > > > > > > > Let me know if that seems like a good compromise. > > > > > > > > - Anton > > > > > > > > > > > > On Apr 25, 2023, at 8:01 PM, Walaa Eldin Moustafa < > > wa.moust...@gmail.com> > > > > wrote: > > > > > > > > To elaborate on LinkedIn's use case: > > > > > > > > * LinkedIn maintains its own fork, but we would like to keep it as > > close > > > > to upstream as possible. > > > > * +1 to Manu on migrations in large companies could take well beyond 18 > > > > months, and it is unlikely to migrate/upgrade more frequently. > > > > * One important use case for the Spark 3.1 module is not necessarily > > > > fixing issues in the module itself, but fixing issues in other core > > > > modules, and having a release that contains core fixes as well as > > Spark 3.1. > > > > * That said, in the last 6 months, there have been 29 commits to Spark > > 3.1 > > > > module, 50 commits to Spark 3.2 module, and 90 to Spark 3.3, in the > > Iceberg > > > > master branch. It seems that Spark 3.1 is reasonably active. > > > > > > > > What does marking as deprecated entail in terms of deleting the code? > > > > Would the guideline be to use 3.2 or 3.3 as an alternative? > > > > > > > > Thanks, > > > > Walaa. > > > > > > > > > > > > > > > > On Tue, Apr 25, 2023 at 12:08 PM Anton Okolnychyi < > > > > aokolnyc...@apple.com.invalid> wrote: > > > > > > > >> Ok, seems like we are in agreement to deprecate 3.1. I’ll fire a PR > > > >> shortly. > > > >> > > > >> Does anyone want to go through changes in 3.3 and 3.2 and find what we > > > >> missed to cherry-pick so that we have that list in one place (e.g. > > create > > > >> an issue)? > > > >> > > > >> Any thoughts on how to mark changes as candidates for cherry-picking? > > > >> Creating an issue? > > > >> > > > >> - Anton > > > >> > > > >> On Apr 24, 2023, at 10:01 AM, Edgar Rodriguez < > > > >> edgar.rodrig...@airbnb.com.INVALID> wrote: > > > >> > > > >> Hi all, > > > >> > > > >> Thanks for the discussion. Similarly to Manu, we're in Spark 3.1.1 and > > > >> Iceberg 1.1.0 - we backport Spark 3.1.1 fixes internally as well. > > It's a > > > >> bit more complicated to move fast on Spark versions internally, > > mainly due > > > >> to the number of scala customers that we have. > > > >> > > > >> I understand maintaining yet another Spark version is burdensome so > > > >> I'm +1 on marking 3.1 deprecated, and I'd be happy to contribute on > > > >> backports if needed on a community maintained branch, we'd just need > > to tag > > > >> changes that may need a backport. > > > >> > > > >> Cheers, > > > >> > > > >> On Sun, Apr 23, 2023 at 4:40 PM Ryan Blue <b...@tabular.io> wrote: > > > >> > > > >>> Thank you for stepping up and offering to help, Manu. I'm glad that > > > >>> you're willing to help with backports. > > > >>> > > > >>> On Sun, Apr 23, 2023 at 2:05 AM Manu Zhang <owenzhang1...@gmail.com> > > > >>> wrote: > > > >>> > > > >>>> You would just end up backporting twice. > > > >>>> > > > >>>> > > > >>>> That's why I said a community maintained branch benefits us, saving > > one > > > >>>> backport. Note the first backport is more difficult, sometimes > > requiring > > > >>>> rewriting the PR since there would be API differences between Spark > > > >>>> versions. > > > >>>> The second backport will be much easier if we focus on bug fixes. > > > >>>> Meanwhile, it's also easier for us to upgrade to Iceberg 1.2+ if 3.1 > > > >>>> support is still available although deprecated. > > > >>>> > > > >>>> > > > >>> > > > >>> -- > > > >>> Ryan Blue > > > >>> Tabular > > > >>> > > > >> > > > >> > > > >> -- > > > >> Edgar R > > > >> > > > >> > > > >> > > > > > > > > > > > > -- > Ryan Blue > Tabular >
