I'm wondering if it is too late to request just one more release of Iceberg with Spark 3.4 support? I have a PR open https://github.com/apache/iceberg/pull/13786 that addresses a whole host of issues with using Comet and Iceberg. This completes the work started by Huaxin in https://github.com/apache/iceberg/pull/13378. At the moment Comet still supports Spark 3.4 and the PR accordingly updates the Spark 3.4 module as well. If we deprecate Spark 3.4, users of Spark 3.4 who want to use Comet will have to build their own. It's not the end of the world, Comet already provides a patch and instructions on how to do this. It is, however, less than ideal. I know that voting has already started (and is practically over) on this issue, but I figured it doesn't hurt to try.
Parth On Tue, Oct 7, 2025 at 7:49 AM Kevin Liu <[email protected]> wrote: > The PR has been merged (https://github.com/apache/iceberg/pull/14099), > and I can see the change reflected on the website, > https://iceberg.apache.org/multi-engine-support/#apache-spark > Thanks everyone for the review. > > I'll start a separate thread for removing spark 3.4 from the repo. > > Best, > Kevin Liu > > > > On Mon, Oct 6, 2025 at 3:03 PM Kevin Liu <[email protected]> wrote: > >> Hey folks, >> >> All (except 1) recent contributors to spark/v3.4 have responded on the >> slack thread above and support removing spark 3.4. The last contributor has >> not responded yet. >> >> If there are no objections, I plan to merge this PR to change the status >> for Spark 3.4 from "Maintained" -> "Deprecated" . >> https://github.com/apache/iceberg/pull/14099 >> Thanks Eduard for the review. >> >> Best, >> Kevin Liu >> >> On Mon, Sep 29, 2025 at 9:52 AM Kevin Liu <[email protected]> wrote: >> >>> Started a slack thread, >>> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1759163920874249 >>> Based on the recently commit activities, I messaged (on slack and pr) 4 >>> contributors who are actively backporting Spark 3.4. >>> >>> Best, >>> Kevin Liu >>> >>> On Mon, Sep 29, 2025 at 9:32 AM Kevin Liu <[email protected]> wrote: >>> >>>> Bumping this thread. >>>> Since there's consensus to mark Spark 3.4 as deprecated, let's merge >>>> this PR to change the status on the docs, >>>> https://github.com/apache/iceberg/pull/14099 >>>> I'll start a new thread for the decision to mark Spark 3.4 as >>>> end-of-life and for removal from the codebase. >>>> >>>> Best, >>>> Kevin Liu >>>> >>>> On Mon, Sep 22, 2025 at 11:13 AM Steve Loughran >>>> <[email protected]> wrote: >>>> >>>>> making hadoop 3.4.1 the base hadoop version has sort of made iceberg >>>>> incompatible with spark 3.4, it just doesn't surface very much. >>>>> >>>>> My PR, https://github.com/apache/iceberg/pull/10233 , which uses a >>>>> hadoop 3.4.1 BulkDelete call not in the spark 3.4 classpath flags this: >>>>> the >>>>> spark 3.4 tests fail with classpath errors. >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, 19 Sept 2025 at 22:02, Kevin Liu <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks Anton and Eduard. I'm ok with being more aggressive with the >>>>>> deprecation schedule. >>>>>> Looking at the git history for `spark/v3.4/` >>>>>> <https://github.com/apache/iceberg/commits/main/spark/v3.4>, there >>>>>> are 5 new commits since the 1.10 release. Only 1 commit (3bbdee9 >>>>>> <https://github.com/apache/iceberg/commit/3bbdee97b0f33794df091c3292de8d54a05dcc1e>) >>>>>> is to backport specifically for Spark 3.4. >>>>>> >>>>>> From Engine Version Lifecycle >>>>>> <https://iceberg.apache.org/multi-engine-support/#engine-version-lifecycle> >>>>>> , >>>>>> > Deprecated: an engine version is no longer actively maintained. >>>>>> People who are still interested in the version can backport any necessary >>>>>> feature or bug fix from newer versions, but the community will not spend >>>>>> effort in achieving feature parity. Iceberg recommends users to move >>>>>> towards a newer version. Contributions to a deprecated version is >>>>>> expected >>>>>> to diminish over time, so that eventually no change is added to a >>>>>> deprecated version. >>>>>> and >>>>>> > End-of-life: a vote can be initiated in the community to fully >>>>>> remove a deprecated version out of the Iceberg repository to mark as its >>>>>> end of life. >>>>>> >>>>>> Let's change the status for Spark 3.4 to "Deprecated". I have >>>>>> prepared a PR already https://github.com/apache/iceberg/pull/14099 >>>>>> >>>>>> I can start another vote thread for removal of Spark 3.4 in the >>>>>> upcoming 1.11 release and mark it as "End-of-life". I already prepared a >>>>>> PR >>>>>> to fully remove Spark 3.4 from the codebase, >>>>>> https://github.com/apache/iceberg/pull/14122 >>>>>> >>>>>> Best, >>>>>> Kevin Liu >>>>>> >>>>>> >>>>>> On Fri, Sep 19, 2025 at 11:29 AM Eduard Tudenhöfner < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I agree with Anton and I would be in favor of just removing it in >>>>>>> the next release. By updating the docs now we can already signal >>>>>>> immediately that Spark 3.4 is deprecated and people can always use >>>>>>> Iceberg >>>>>>> 1.10 when needing Spark 3.4 support. >>>>>>> >>>>>>> On Fri, Sep 19, 2025 at 7:06 PM Anton Okolnychyi < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I know we followed this rule of deprecating a Spark version in one >>>>>>>> release and then removing it in the next one. Shall we ask ourselves >>>>>>>> whether it is still the model we want to follow? >>>>>>>> >>>>>>>> My problem like before is that we release a new Iceberg jar that is >>>>>>>> supposed to contain the latest and greatest features but the >>>>>>>> functionality >>>>>>>> for older Spark versions is severely lagging. >>>>>>>> >>>>>>>> We initially kept older Spark modules in main to give folks in the >>>>>>>> community a place to maintain these older integrations and >>>>>>>> collaborate. I >>>>>>>> don’t see a lot of interest in that, if I am being honest. Instead, it >>>>>>>> became a liability for devs as all major format features now have to >>>>>>>> work >>>>>>>> with those old Spark integrations. It is hurting the velocity of the >>>>>>>> project and recent row ID work is an example of that. >>>>>>>> >>>>>>>> - Anton >>>>>>>> >>>>>>>> On Fri, Sep 19, 2025 at 6:33 PM Kevin Liu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> > why not just remove Spark 3.4 for the next 1.11 release? Or do >>>>>>>>> we usually wait for one more release and remove it in the 1.12 release >>>>>>>>> after marking 3.4 as deprecated in the engine status doc page? >>>>>>>>> >>>>>>>>> My preference is to mark as deprecated for one release and remove >>>>>>>>> in the following. >>>>>>>>> >>>>>>>>> To quote JB: >>>>>>>>> "announce" the deprecation in 1.11 and remove 1.12, it gives >>>>>>>>> time for users to "adapt". >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Kevin Liu >>>>>>>>> >>>>>>>>> On Fri, Sep 19, 2025 at 9:26 AM Steven Wu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Following up on Manu's question, why not just remove Spark 3.4 >>>>>>>>>> for the next 1.11 release? Or do we usually wait for one more >>>>>>>>>> release and >>>>>>>>>> remove it in the 1.12 release after marking 3.4 as deprecated in the >>>>>>>>>> engine >>>>>>>>>> status doc page? >>>>>>>>>> >>>>>>>>>> On Fri, Sep 19, 2025 at 9:12 AM Kevin Liu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Given the many +1's here, I've moved the PR to deprecate 3.4 to >>>>>>>>>>> "ready for review", https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>> >>>>>>>>>>> > Does it mean we will stop back-porting PRs to Spark 3.4 for >>>>>>>>>>> 1.11? >>>>>>>>>>> >>>>>>>>>>> Not necessarily. There's a lot of Spark 3.4 backports already, >>>>>>>>>>> https://github.com/apache/iceberg/commits/main/spark/v3.4 >>>>>>>>>>> I suggest we continue to backport for consistency and then stop >>>>>>>>>>> right after the 1.11 release. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Kevin Liu >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 19, 2025 at 6:18 AM Amogh Jahagirdar < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 19, 2025 at 2:03 AM Péter Váry < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> Eduard Tudenhöfner <[email protected]> ezt írta >>>>>>>>>>>>> (időpont: 2025. szept. 19., P, 8:56): >>>>>>>>>>>>> >>>>>>>>>>>>>> +1 on deprecating Spark 3.4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Sep 18, 2025 at 8:36 AM Steve < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 22:52 Jean-Baptiste Onofré < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I agree about the plan to "announce" the deprecation in >>>>>>>>>>>>>>>> 1.11 and >>>>>>>>>>>>>>>> remove 1.12, it gives time for users to "adapt". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 10:31 PM Kevin Liu < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Hi everyone, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > I’d like to bring up the topic of deprecating Spark 3.4 >>>>>>>>>>>>>>>> in an upcoming release. Anton initially suggested this during >>>>>>>>>>>>>>>> our previous >>>>>>>>>>>>>>>> dev list discussion about maintaining feature parity across >>>>>>>>>>>>>>>> the Spark >>>>>>>>>>>>>>>> versions we support for 1.10. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Currently, we support two different Spark 3.x versions, >>>>>>>>>>>>>>>> 3.4 and 3.5. Spark 3.4’s last maintenance release was in >>>>>>>>>>>>>>>> October 2024, and >>>>>>>>>>>>>>>> it is now considered end-of-life. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > What are your thoughts on marking Spark 3.4 as deprecated >>>>>>>>>>>>>>>> in 1.11 and removing it in 1.12? >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > For reference, here's the previous discussion thread on >>>>>>>>>>>>>>>> deprecating Spark 3.3. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Best, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Kevin Liu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
