For the 3.1 activities that Ryan linked, 3.1 are updated probably for the requirement of backporting (keeping 3.1, 3.2, 3.3 in sync). It is the adopted policy. Not sure if it is an indication that people are actively collaborating on 3.1.
As Anton was saying, backporting/syncing 4 versions (3.1, 3.2, 3.3, 3.4) is a pretty high budden. On Fri, Apr 21, 2023 at 2:29 PM Anton Okolnychyi <aokolnyc...@apple.com.invalid> wrote: > If it is being used by folks in the community, let’s keep it for now. That > said, let’s come up with a strategy on when to eventually drop it as the > list cannot grow indefinitely. Our initial agreement was to keep last 3 > (except Spark LTS versions), which worked well for 18 months of support > promised by the Spark community. At this point, Spark will not release any > bug fixes for 3.1, even critical. > > Walaa, Edgar, can you tell us a little bit about the Spark 3.1 integration > you depend on? Do you have your own Iceberg/Spark forks? Is an updated > Iceberg core module the primary thing you are looking for? How do you > deal with Spark bugs? > > My biggest worry is that our Spark 3.1 integration randomly gets some > updates from time to time. By releasing those jars with each Iceberg > version, we send a message that it is being actively maintained and worked > on. That’s actually not true, we cherry-pick only some changes. Also, it is > still part of our release cycle, so it must be checked and tested (our next > release will have 3.1, 3.2, 3.3 and 3.4 integrations to test). > > I am going to close the PR for now but it would be great to find a good > way to handle this in the future. At least, we have to document what kind > of expectations our users should have. Do we promise that all bug fixes > discovered in newer Spark versions will be cherry-picked to all older Spark > versions? I am not sure that’s true at this point. > > - Anton > > > On Apr 21, 2023, at 10:29 AM, Ryan Blue <b...@tabular.io> wrote: > > According to Spark docs, a minor release will be supported for 18 months > and 3.1 was released 2021-03-02, more than 2 years ago. I don't think we > should expect any further updates from the Spark community for the 3.1 line. > > I'm also not sure that there is a problem continuing to release Iceberg's > module for 3.1. It is still being updated > <https://github.com/apache/iceberg/commits/master/spark/v3.1> and I don't > think it is preventing us from continuing work on the later versions. Makes > sense to me to keep it if people are collaborating there. We should > evaluate this again soon though. > > On Fri, Apr 21, 2023 at 8:27 AM Edgar Rodriguez < > edgar.rodrig...@airbnb.com.invalid> wrote: > >> Airbnb is also still on Spark 3.1 and I echo some of Walaa's comments. >> >> Cheers, >> >> On Thu, Apr 20, 2023 at 8:14 PM Walaa Eldin Moustafa < >> wa.moust...@gmail.com> wrote: >> >>> LinkedIn is still on Spark 3.1. I am guessing a number of other >>> companies could be in the same boat. I feel the argument for Spark 2.4 is >>> different from that of Spark 3.1 and it would be great if we can continue >>> to support 3.1 for some time. >>> >>> On Wed, Apr 19, 2023 at 11:06 AM Ryan Blue <b...@tabular.io> wrote: >>> >>>> +1 >>>> >>>> As we said in the 2.4 discussion, the format itself should provide >>>> forward compatibility with tables and it is more clear that we aren't >>>> adding new features if you have to use older versions for Spark 3.1. >>>> >>>> On Wed, Apr 19, 2023 at 10:08 AM Anton Okolnychyi < >>>> aokolnyc...@apple.com.invalid> wrote: >>>> >>>>> Hey folks, >>>>> >>>>> What does everybody think about Spark 3.1 support after we add Spark >>>>> 3.4 support? Our initial plan was to release jars for the last 3 versions. >>>>> Are there any blockers for dropping 3.1? >>>>> >>>>> - Anton >>>> >>>> >>>> >>>> -- >>>> Ryan Blue >>>> Tabular >>>> >>> >> >> -- >> Edgar R >> > > > -- > Ryan Blue > Tabular > > >
