I am planning to cut an RC tomorrow. I'll submit a PR to drop 3.1 later today.

On 2023/09/25 03:10:14 Anton Okolnychyi wrote:
> Let's wait until we are ready to cut an RC to see if anyone still expresses 
> interest in Spark 3.1. If not, I propose to not include it in 1.4 as it has 
> been already deprecated when we released 1.3.
> 
> On 2023/09/21 09:07:47 Eduard Tudenhoefner wrote:
> > +1 for removing Spark 3.1 and deprecating Spark 3.2
> > 
> > On Thu, Sep 21, 2023 at 10:44 AM Jean-Baptiste Onofré <j...@nanthrax.net>
> > wrote:
> > 
> > > Hi Anton,
> > >
> > > imho for 1.4.0, we can deprecate/remove Spark 3.1 support. As it's
> > > major release, we can remove old version support. Spark 3.1 users can
> > > still use Iceberg 1.3.x.
> > >
> > > That's why I proposed a LTS policy for our users, I will come with a
> > > proposal about that.
> > >
> > > Thanks !
> > > Regards
> > > JB
> > >
> > > On Wed, Sep 20, 2023 at 11:53 PM Anton Okolnychyi
> > > <aokolnyc...@apache.org> wrote:
> > > >
> > > > Just checking in to see how we feel about Spark 3.1 now. We support 5
> > > different Spark versions at this point: 3.1, 3.2, 3.3, 3.4 and 3.5. Any
> > > thoughts on making Iceberg 1.4 the last release with Spark 3.1?
> > > >
> > > > On 2023/04/27 04:58:06 Walaa Eldin Moustafa wrote:
> > > > > Yes, that sounds like a good compromise. Initially I was looking at
> > > > > deprecation guidelines in [1], but I see you are referring to [2].
> > > > >
> > > > > [1] https://iceberg.apache.org/contribute/
> > > > > [2]
> > > > >
> > > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status
> > > > >
> > > > > On Wed, Apr 26, 2023 at 8:10 AM Anton Okolnychyi
> > > > > <aokolnyc...@apple.com.invalid> wrote:
> > > > >
> > > > > > Got it. Given that quite a bit of folks still use 3.1, I don’t think
> > > we
> > > > > > would remove it unless the branch becomes inactive. Marking it as
> > > > > > deprecated would allow us to indicate that it may not be as
> > > up-to-date and
> > > > > > complete as other versions and some performance enhancements or even
> > > minor
> > > > > > bug fixes may not be there. That would solve one of the concerns I
> > > raised
> > > > > > earlier. We should discourage users from onboarding new use cases on
> > > 3.1.
> > > > > >
> > > > > > I believe our doc summarizes the message pretty well.
> > > > > >
> > > > > >
> > > > > >    1. *Deprecated*: an engine version is no longer actively
> > > maintained.
> > > > > >    People who are still interested in the version can backport any
> > > necessary
> > > > > >    feature or bug fix from newer versions, but the community will
> > > not spend
> > > > > >    effort in achieving feature parity. Iceberg recommends users to
> > > move
> > > > > >    towards a newer version. Contributions to a deprecated version is
> > > expected
> > > > > >    to diminish over time, so that eventually no change is added to a
> > > > > >    deprecated version.
> > > > > >
> > > > > >
> > > > > >
> > > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status
> > > > > >
> > > > > > Let me know if that seems like a good compromise.
> > > > > >
> > > > > > - Anton
> > > > > >
> > > > > >
> > > > > > On Apr 25, 2023, at 8:01 PM, Walaa Eldin Moustafa <
> > > wa.moust...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > To elaborate on LinkedIn's use case:
> > > > > >
> > > > > > * LinkedIn maintains its own fork, but we would like to keep it as
> > > close
> > > > > > to upstream as possible.
> > > > > > * +1 to Manu on migrations in large companies could take well beyond
> > > 18
> > > > > > months, and it is unlikely to migrate/upgrade more frequently.
> > > > > > * One important use case for the Spark 3.1 module is not necessarily
> > > > > > fixing issues in the module itself, but fixing issues in other core
> > > > > > modules, and having a release that contains core fixes as well as
> > > Spark 3.1.
> > > > > > * That said, in the last 6 months, there have been 29 commits to
> > > Spark 3.1
> > > > > > module, 50 commits to Spark 3.2 module, and 90 to Spark 3.3, in the
> > > Iceberg
> > > > > > master branch. It seems that Spark 3.1 is reasonably active.
> > > > > >
> > > > > > What does marking as deprecated entail in terms of deleting the 
> > > > > > code?
> > > > > > Would the guideline be to use 3.2 or 3.3 as an alternative?
> > > > > >
> > > > > > Thanks,
> > > > > > Walaa.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 25, 2023 at 12:08 PM Anton Okolnychyi <
> > > > > > aokolnyc...@apple.com.invalid> wrote:
> > > > > >
> > > > > >> Ok, seems like we are in agreement to deprecate 3.1. I’ll fire a PR
> > > > > >> shortly.
> > > > > >>
> > > > > >> Does anyone want to go through changes in 3.3 and 3.2 and find what
> > > we
> > > > > >> missed to cherry-pick so that we have that list in one place (e.g.
> > > create
> > > > > >> an issue)?
> > > > > >>
> > > > > >> Any thoughts on how to mark changes as candidates for
> > > cherry-picking?
> > > > > >> Creating an issue?
> > > > > >>
> > > > > >> - Anton
> > > > > >>
> > > > > >> On Apr 24, 2023, at 10:01 AM, Edgar Rodriguez <
> > > > > >> edgar.rodrig...@airbnb.com.INVALID> wrote:
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >> Thanks for the discussion. Similarly to Manu, we're in Spark 3.1.1
> > > and
> > > > > >> Iceberg 1.1.0 - we backport Spark 3.1.1 fixes internally as well.
> > > It's a
> > > > > >> bit more complicated to move fast on Spark versions internally,
> > > mainly due
> > > > > >> to the number of scala customers that we have.
> > > > > >>
> > > > > >> I understand maintaining yet another Spark version is burdensome so
> > > > > >> I'm +1 on marking 3.1 deprecated, and I'd be happy to contribute on
> > > > > >> backports if needed on a community maintained branch, we'd just
> > > need to tag
> > > > > >> changes that may need a backport.
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> On Sun, Apr 23, 2023 at 4:40 PM Ryan Blue <b...@tabular.io> wrote:
> > > > > >>
> > > > > >>> Thank you for stepping up and offering to help, Manu. I'm glad 
> > > > > >>> that
> > > > > >>> you're willing to help with backports.
> > > > > >>>
> > > > > >>> On Sun, Apr 23, 2023 at 2:05 AM Manu Zhang <
> > > owenzhang1...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>>   You would just end up backporting twice.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> That's why I said a community maintained branch benefits us,
> > > saving one
> > > > > >>>> backport. Note the first backport is more difficult, sometimes
> > > requiring
> > > > > >>>> rewriting the PR since there would be API differences between
> > > Spark
> > > > > >>>> versions.
> > > > > >>>> The second backport will be much easier if we focus on bug fixes.
> > > > > >>>> Meanwhile, it's also easier for us to upgrade to Iceberg 1.2+ if
> > > 3.1
> > > > > >>>> support is still available although deprecated.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Ryan Blue
> > > > > >>> Tabular
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Edgar R
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > >
> > 
> 

Reply via email to