https://github.com/apache/spark/pull/58929 If I understand correctly about the issue, this is a blocker for 4.2 as it can break checkpoint in stateful workloads (despite that v2 is not the default format). The issue isn't framed well enough to emphasize the severity; once we have a new JIRA ticket (or rewritten JIRA ticket), I'll bump the severity to blocker.
On Thu, Oct 1, 2026 at 12:29 PM Szehon Ho <[email protected]> wrote: > I took a look at the recent 4.3 fixes/blockers. We fixed two for 4.2, but > we decided not to do the other two for 4.2, as listed below. > > - SPARK-59804 <https://issues.apache.org/jira/browse/SPARK-59804> — PR > #59074 <https://github.com/apache/spark/pull/59074> — *merged into 4.2* > - SPARK-51830 <https://issues.apache.org/jira/browse/SPARK-51830> — PR > #57994 <https://github.com/apache/spark/pull/57994>; *merged into > 4.2* > - SPARK-44462 <https://issues.apache.org/jira/browse/SPARK-44462> — PR > #55410 <https://github.com/apache/spark/pull/55410> — took a look. it > is a complex fix with some risk, and there's some behavior change. We are > leaning towards omitting it from 4.2.1. > - SPARK-59122 <https://issues.apache.org/jira/browse/SPARK-59122> — PR > #58419 <https://github.com/apache/spark/pull/58419> — took a look. It > is also not so simple and also UNION codegen is disabled on 4.2, so also > leaning towards omitting it from 4.2.1. > > I am not sure we need to wait until 4.3 RC passes, as the scope of 4.2.1 > maintenance release should be a more focused fix for 4.2 users hitting > obvious issues (example, on Iceberg side). As we see from some examples, > some 4.3 issues may not be appropriate for 4.2. I would prefer to restart > the process and halt only if there is an unresolved 4.2 blocker, unless > someone feels strongly otherwise. > > Thanks, > Szehon > > On Mon, Sep 28, 2026 at 7:01 PM Holden Karau <[email protected]> > wrote: > >> I think it might be good to hold the point releases until the 4.3 RC >> passes just incase we’ve got any blockers we discover and want to backport? >> I don’t think it’s a “must” hold just it might make life easier. >> >> >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >> On Mon, Sep 28, 2026 at 6:20 PM Uroš Bojanić <[email protected]> wrote: >> >>> Hi folks, thank you for working on this! Please also let me know if any >>> of the issues above are blockers for other releases (e.g. 4.1.4, 4.0.5) or >>> not; there are separate threads for those releases as well. >>> >>> On 2026/09/28 09:11:21 [email protected] wrote: >>> > Please vote on releasing the following candidate as Apache Spark >>> version 4.2.1. >>> > >>> > The vote is open until Thu, 01 Oct 2026 03:11:20 PDT and passes if a >>> majority +1 PMC votes are cast, with >>> > a minimum of 3 +1 votes. >>> > >>> > [ ] +1 Release this package as Apache Spark 4.2.1 >>> > [ ] -1 Do not release this package because ... >>> > >>> > To learn more about Apache Spark, please see https://spark.apache.org/ >>> > >>> > The tag to be voted on is v4.2.1-rc1 (commit 72c18760854): >>> > https://github.com/apache/spark/tree/v4.2.1-rc1 >>> > >>> > The release files, including signatures, digests, etc. can be found at: >>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/ >>> > >>> > Signatures used for Spark RCs can be found in this file: >>> > https://downloads.apache.org/spark/KEYS >>> > >>> > The staging repository for this release can be found at: >>> > >>> https://repository.apache.org/content/repositories/orgapachespark-1531/ >>> > >>> > The documentation corresponding to this release can be found at: >>> > https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-docs/ >>> > >>> > The list of bug fixes going into 4.2.1 can be found at the following >>> URL: >>> > https://issues.apache.org/jira/projects/SPARK/versions/12357191 >>> > >>> > FAQ >>> > >>> > ========================= >>> > How can I help test this release? >>> > ========================= >>> > >>> > If you are a Spark user, you can help us test this release by taking >>> > an existing Spark workload and running on this release candidate, then >>> > reporting any regressions. >>> > >>> > If you're working in PySpark you can set up a virtual env and install >>> > the current RC via "pip install >>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/pyspark-4.2.1.tar.gz >>> " >>> > and see if anything important breaks. >>> > In the Java/Scala, you can add the staging repository to your >>> project's resolvers and test >>> > with the RC (make sure to clean up the artifact cache before/after so >>> > you don't end up building with an out of date RC going forward). >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe e-mail: [email protected] >>> > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>>
