Correct, the proposal is to initially deprecate but still release whatever the community cherry-picks to that branch. Releasing without the deprecation signals that the dependency is actively maintained while it is not. The 3.2 integration significantly lags behind and does not represent the latest and most capable integration with Spark.
I've raised a separate question about 3.1. On 2023/09/21 00:06:35 Pucheng Yang wrote: > Thanks! > > On Wed, Sep 20, 2023 at 5:03 PM Ryan Blue <b...@tabular.io> wrote: > > > Ah, right. I actually was talking about if we removed 3.2 from the source > > tree. But you're right. If we just deprecated it, you'd still be able to > > backport things to it until we actually do remove it. > > > > On Wed, Sep 20, 2023 at 5:02 PM Pucheng Yang <py...@pinterest.com.invalid> > > wrote: > > > >> Got it, so "deprecate spark 3.2 support" does not mean removing the spark > >> 3.2 module in the Iceberg project right? > >> > >> And it also means maybe some new changes will only be available on Spark > >> 3.3 +, and such change will likely not get backported. to <3.3 ? But if we > >> are interested, we can do that ourselves, right? > >> > >> Thanks > >> > >> On Wed, Sep 20, 2023 at 4:57 PM Ryan Blue <b...@tabular.io> wrote: > >> > >>> Pucheng, you can continue using older releases of Iceberg or you may > >>> already have your own fork. The older versions will continue to work and > >>> we > >>> can still do patch releases for the point versions if there are serious > >>> issues to fix. > >>> > >>> On Wed, Sep 20, 2023 at 4:35 PM Pucheng Yang <py...@pinterest.com.invalid> > >>> wrote: > >>> > >>>> Like Linkedin (mentioned in another thread), Pinterest is on Spark 3.2 > >>>> and there is no immediate plan to upgrade to the new Spark version as the > >>>> migration cost is very high and the process is slow. > >>>> > >>>> What will be the implications of not having Spark-3.2 module in Iceberg > >>>> any more? Based on your past experience, how do companies move forward to > >>>> maintain Iceberg with removed Spark versions? Do those companies own > >>>> their > >>>> enhancements or fixes internally? > >>>> > >>>> Thanks! > >>>> > >>>> On Wed, Sep 20, 2023 at 4:00 PM Ryan Blue <b...@tabular.io> wrote: > >>>> > >>>>> +1 > >>>>> > >>>>> On Wed, Sep 20, 2023 at 3:02 PM Anton Okolnychyi > >>>>> <aokolnyc...@apple.com.invalid> wrote: > >>>>> > >>>>>> Shall we consider deprecating our Spark 3.2 support? That Spark > >>>>>> version is no longer being maintained by the Spark community and is not > >>>>>> under active development in Iceberg. It was released in October, 2021 > >>>>>> and > >>>>>> passed the 18 month maintenance mark in Spark. > >>>>>> > >>>>>> - Anton > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Ryan Blue > >>>>> Tabular > >>>>> > >>>> > >>> > >>> -- > >>> Ryan Blue > >>> Tabular > >>> > >> > > > > -- > > Ryan Blue > > Tabular > > >
