To elaborate on LinkedIn's use case: * LinkedIn maintains its own fork, but we would like to keep it as close to upstream as possible. * +1 to Manu on migrations in large companies could take well beyond 18 months, and it is unlikely to migrate/upgrade more frequently. * One important use case for the Spark 3.1 module is not necessarily fixing issues in the module itself, but fixing issues in other core modules, and having a release that contains core fixes as well as Spark 3.1. * That said, in the last 6 months, there have been 29 commits to Spark 3.1 module, 50 commits to Spark 3.2 module, and 90 to Spark 3.3, in the Iceberg master branch. It seems that Spark 3.1 is reasonably active.
What does marking as deprecated entail in terms of deleting the code? Would the guideline be to use 3.2 or 3.3 as an alternative? Thanks, Walaa. On Tue, Apr 25, 2023 at 12:08 PM Anton Okolnychyi <aokolnyc...@apple.com.invalid> wrote: > Ok, seems like we are in agreement to deprecate 3.1. I’ll fire a PR > shortly. > > Does anyone want to go through changes in 3.3 and 3.2 and find what we > missed to cherry-pick so that we have that list in one place (e.g. create > an issue)? > > Any thoughts on how to mark changes as candidates for cherry-picking? > Creating an issue? > > - Anton > > On Apr 24, 2023, at 10:01 AM, Edgar Rodriguez < > edgar.rodrig...@airbnb.com.INVALID> wrote: > > Hi all, > > Thanks for the discussion. Similarly to Manu, we're in Spark 3.1.1 and > Iceberg 1.1.0 - we backport Spark 3.1.1 fixes internally as well. It's a > bit more complicated to move fast on Spark versions internally, mainly due > to the number of scala customers that we have. > > I understand maintaining yet another Spark version is burdensome so I'm +1 > on marking 3.1 deprecated, and I'd be happy to contribute on backports if > needed on a community maintained branch, we'd just need to tag changes that > may need a backport. > > Cheers, > > On Sun, Apr 23, 2023 at 4:40 PM Ryan Blue <b...@tabular.io> wrote: > >> Thank you for stepping up and offering to help, Manu. I'm glad that >> you're willing to help with backports. >> >> On Sun, Apr 23, 2023 at 2:05 AM Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> You would just end up backporting twice. >>> >>> >>> That's why I said a community maintained branch benefits us, saving one >>> backport. Note the first backport is more difficult, sometimes requiring >>> rewriting the PR since there would be API differences between Spark >>> versions. >>> The second backport will be much easier if we focus on bug fixes. >>> Meanwhile, it's also easier for us to upgrade to Iceberg 1.2+ if 3.1 >>> support is still available although deprecated. >>> >>> >> >> -- >> Ryan Blue >> Tabular >> > > > -- > Edgar R > > >
