Instead of removing Spark 3.4 in 1.11, could we keep it for a couple more versions?
Thanks, Huaxin On Mon, Oct 13, 2025 at 9:57 AM Parth Chandra <[email protected]> wrote: > I'm wondering if it is too late to request just one more release of > Iceberg with Spark 3.4 support? I have a PR open > https://github.com/apache/iceberg/pull/13786 that addresses a whole host > of issues with using Comet and Iceberg. This completes the work started by > Huaxin in https://github.com/apache/iceberg/pull/13378. > At the moment Comet still supports Spark 3.4 and the PR accordingly > updates the Spark 3.4 module as well. If we deprecate Spark 3.4, users of > Spark 3.4 who want to use Comet will have to build their own. It's not the > end of the world, Comet already provides a patch and instructions on how to > do this. It is, however, less than ideal. > I know that voting has already started (and is practically over) on this > issue, but I figured it doesn't hurt to try. > > Parth > > On Tue, Oct 7, 2025 at 7:49 AM Kevin Liu <[email protected]> wrote: > >> The PR has been merged (https://github.com/apache/iceberg/pull/14099), >> and I can see the change reflected on the website, >> https://iceberg.apache.org/multi-engine-support/#apache-spark >> Thanks everyone for the review. >> >> I'll start a separate thread for removing spark 3.4 from the repo. >> >> Best, >> Kevin Liu >> >> >> >> On Mon, Oct 6, 2025 at 3:03 PM Kevin Liu <[email protected]> wrote: >> >>> Hey folks, >>> >>> All (except 1) recent contributors to spark/v3.4 have responded on the >>> slack thread above and support removing spark 3.4. The last contributor has >>> not responded yet. >>> >>> If there are no objections, I plan to merge this PR to change the status >>> for Spark 3.4 from "Maintained" -> "Deprecated" . >>> https://github.com/apache/iceberg/pull/14099 >>> Thanks Eduard for the review. >>> >>> Best, >>> Kevin Liu >>> >>> On Mon, Sep 29, 2025 at 9:52 AM Kevin Liu <[email protected]> wrote: >>> >>>> Started a slack thread, >>>> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1759163920874249 >>>> Based on the recently commit activities, I messaged (on slack and pr) 4 >>>> contributors who are actively backporting Spark 3.4. >>>> >>>> Best, >>>> Kevin Liu >>>> >>>> On Mon, Sep 29, 2025 at 9:32 AM Kevin Liu <[email protected]> >>>> wrote: >>>> >>>>> Bumping this thread. >>>>> Since there's consensus to mark Spark 3.4 as deprecated, let's merge >>>>> this PR to change the status on the docs, >>>>> https://github.com/apache/iceberg/pull/14099 >>>>> I'll start a new thread for the decision to mark Spark 3.4 as >>>>> end-of-life and for removal from the codebase. >>>>> >>>>> Best, >>>>> Kevin Liu >>>>> >>>>> On Mon, Sep 22, 2025 at 11:13 AM Steve Loughran >>>>> <[email protected]> wrote: >>>>> >>>>>> making hadoop 3.4.1 the base hadoop version has sort of made iceberg >>>>>> incompatible with spark 3.4, it just doesn't surface very much. >>>>>> >>>>>> My PR, https://github.com/apache/iceberg/pull/10233 , which uses a >>>>>> hadoop 3.4.1 BulkDelete call not in the spark 3.4 classpath flags this: >>>>>> the >>>>>> spark 3.4 tests fail with classpath errors. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, 19 Sept 2025 at 22:02, Kevin Liu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks Anton and Eduard. I'm ok with being more aggressive with the >>>>>>> deprecation schedule. >>>>>>> Looking at the git history for `spark/v3.4/` >>>>>>> <https://github.com/apache/iceberg/commits/main/spark/v3.4>, there >>>>>>> are 5 new commits since the 1.10 release. Only 1 commit (3bbdee9 >>>>>>> <https://github.com/apache/iceberg/commit/3bbdee97b0f33794df091c3292de8d54a05dcc1e>) >>>>>>> is to backport specifically for Spark 3.4. >>>>>>> >>>>>>> From Engine Version Lifecycle >>>>>>> <https://iceberg.apache.org/multi-engine-support/#engine-version-lifecycle> >>>>>>> , >>>>>>> > Deprecated: an engine version is no longer actively maintained. >>>>>>> People who are still interested in the version can backport any >>>>>>> necessary >>>>>>> feature or bug fix from newer versions, but the community will not spend >>>>>>> effort in achieving feature parity. Iceberg recommends users to move >>>>>>> towards a newer version. Contributions to a deprecated version is >>>>>>> expected >>>>>>> to diminish over time, so that eventually no change is added to a >>>>>>> deprecated version. >>>>>>> and >>>>>>> > End-of-life: a vote can be initiated in the community to fully >>>>>>> remove a deprecated version out of the Iceberg repository to mark as its >>>>>>> end of life. >>>>>>> >>>>>>> Let's change the status for Spark 3.4 to "Deprecated". I have >>>>>>> prepared a PR already https://github.com/apache/iceberg/pull/14099 >>>>>>> >>>>>>> I can start another vote thread for removal of Spark 3.4 in the >>>>>>> upcoming 1.11 release and mark it as "End-of-life". I already prepared >>>>>>> a PR >>>>>>> to fully remove Spark 3.4 from the codebase, >>>>>>> https://github.com/apache/iceberg/pull/14122 >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 19, 2025 at 11:29 AM Eduard Tudenhöfner < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I agree with Anton and I would be in favor of just removing it in >>>>>>>> the next release. By updating the docs now we can already signal >>>>>>>> immediately that Spark 3.4 is deprecated and people can always use >>>>>>>> Iceberg >>>>>>>> 1.10 when needing Spark 3.4 support. >>>>>>>> >>>>>>>> On Fri, Sep 19, 2025 at 7:06 PM Anton Okolnychyi < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I know we followed this rule of deprecating a Spark version in one >>>>>>>>> release and then removing it in the next one. Shall we ask ourselves >>>>>>>>> whether it is still the model we want to follow? >>>>>>>>> >>>>>>>>> My problem like before is that we release a new Iceberg jar that >>>>>>>>> is supposed to contain the latest and greatest features but the >>>>>>>>> functionality for older Spark versions is severely lagging. >>>>>>>>> >>>>>>>>> We initially kept older Spark modules in main to give folks in the >>>>>>>>> community a place to maintain these older integrations and >>>>>>>>> collaborate. I >>>>>>>>> don’t see a lot of interest in that, if I am being honest. Instead, it >>>>>>>>> became a liability for devs as all major format features now have to >>>>>>>>> work >>>>>>>>> with those old Spark integrations. It is hurting the velocity of the >>>>>>>>> project and recent row ID work is an example of that. >>>>>>>>> >>>>>>>>> - Anton >>>>>>>>> >>>>>>>>> On Fri, Sep 19, 2025 at 6:33 PM Kevin Liu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> > why not just remove Spark 3.4 for the next 1.11 release? Or do >>>>>>>>>> we usually wait for one more release and remove it in the 1.12 >>>>>>>>>> release >>>>>>>>>> after marking 3.4 as deprecated in the engine status doc page? >>>>>>>>>> >>>>>>>>>> My preference is to mark as deprecated for one release and remove >>>>>>>>>> in the following. >>>>>>>>>> >>>>>>>>>> To quote JB: >>>>>>>>>> "announce" the deprecation in 1.11 and remove 1.12, it gives >>>>>>>>>> time for users to "adapt". >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Kevin Liu >>>>>>>>>> >>>>>>>>>> On Fri, Sep 19, 2025 at 9:26 AM Steven Wu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Following up on Manu's question, why not just remove Spark 3.4 >>>>>>>>>>> for the next 1.11 release? Or do we usually wait for one more >>>>>>>>>>> release and >>>>>>>>>>> remove it in the 1.12 release after marking 3.4 as deprecated in >>>>>>>>>>> the engine >>>>>>>>>>> status doc page? >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 19, 2025 at 9:12 AM Kevin Liu <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Given the many +1's here, I've moved the PR to deprecate 3.4 to >>>>>>>>>>>> "ready for review", >>>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>>> >>>>>>>>>>>> > Does it mean we will stop back-porting PRs to Spark 3.4 for >>>>>>>>>>>> 1.11? >>>>>>>>>>>> >>>>>>>>>>>> Not necessarily. There's a lot of Spark 3.4 backports already, >>>>>>>>>>>> https://github.com/apache/iceberg/commits/main/spark/v3.4 >>>>>>>>>>>> I suggest we continue to backport for consistency and then stop >>>>>>>>>>>> right after the 1.11 release. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Kevin Liu >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:18 AM Amogh Jahagirdar < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 19, 2025 at 2:03 AM Péter Váry < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> +1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Eduard Tudenhöfner <[email protected]> ezt írta >>>>>>>>>>>>>> (időpont: 2025. szept. 19., P, 8:56): >>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 on deprecating Spark 3.4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Sep 18, 2025 at 8:36 AM Steve < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 22:52 Jean-Baptiste Onofré < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I agree about the plan to "announce" the deprecation in >>>>>>>>>>>>>>>>> 1.11 and >>>>>>>>>>>>>>>>> remove 1.12, it gives time for users to "adapt". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 10:31 PM Kevin Liu < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Hi everyone, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > I’d like to bring up the topic of deprecating Spark 3.4 >>>>>>>>>>>>>>>>> in an upcoming release. Anton initially suggested this during >>>>>>>>>>>>>>>>> our previous >>>>>>>>>>>>>>>>> dev list discussion about maintaining feature parity across >>>>>>>>>>>>>>>>> the Spark >>>>>>>>>>>>>>>>> versions we support for 1.10. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Currently, we support two different Spark 3.x versions, >>>>>>>>>>>>>>>>> 3.4 and 3.5. Spark 3.4’s last maintenance release was in >>>>>>>>>>>>>>>>> October 2024, and >>>>>>>>>>>>>>>>> it is now considered end-of-life. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > What are your thoughts on marking Spark 3.4 as >>>>>>>>>>>>>>>>> deprecated in 1.11 and removing it in 1.12? >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > For reference, here's the previous discussion thread on >>>>>>>>>>>>>>>>> deprecating Spark 3.3. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Best, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Kevin Liu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>
