W.r.t. blockers for Apache Spark 4.3.0 - IMHO we can't always decide the backport for the blocker which is complicated enough to potentially trigger another kind of issue. Users expect the upgrade of maintenance releases to be mostly straightforward and painless; unless it's directly related to the correctness, it'd be better to evaluate the risk of backport even though it's marked as a blocker in some sense.
On Tue, Sep 29, 2026 at 9:38 AM Szehon Ho <[email protected]> wrote: > Thanks, Holden and Liang-Chi. I’ll hold off on finalizing RC1 while we > coordinate the relevant fixes for 4.2.1: > > - *Raised on the 4.2.1 RC1 vote:* SPARK-59804 > <https://issues.apache.org/jira/browse/SPARK-59804> / #59074 > <https://github.com/apache/spark/pull/59074> — event-log replay > regressions introduced by SPARK-59407, which is included in RC1. > - *Explicit blocker in the 4.3.0 RC1 vote:* SPARK-44462 > <https://issues.apache.org/jira/browse/SPARK-44462> / #55410 > <https://github.com/apache/spark/pull/55410> — incorrect session used > by Spark Connect foreachBatch; also affects 4.2. > - *Open JIRA blocker affecting 4.3.0:* SPARK-59122 > <https://issues.apache.org/jira/browse/SPARK-59122> / #58419 > <https://github.com/apache/spark/pull/58419> — UNION > correctness/codegen issues. The affected code exists in 4.2, although UNION > codegen is off by default there. > - *Additional backport candidate from the JIRA check:* SPARK-51830 > <https://issues.apache.org/jira/browse/SPARK-51830> / #57994 > <https://github.com/apache/spark/pull/57994> — Critical-priority > partition-type validation bug, already fixed in 4.3 but missing from 4.2; > not explicitly raised as a release blocker. > > Let me know if additional fixes are needed. > > Also, thanks Huaxin for checking the signing key. It is already in > dev/spark/KEYS <https://dist.apache.org/repos/dist/dev/spark/KEYS>, but > is missing from the published KEYS linked in the vote email, as I need a > PMC member to help add it to release/spark/KEYS > <https://dist.apache.org/repos/dist/release/spark/KEYS>. > > Thanks, > Szehon > > On Mon, Sep 28, 2026 at 10:43 AM Holden Karau <[email protected]> > wrote: > >> Do we want to wait for 4.3.0 to go out so we know we've got all of the >> fixes backported from 4.3 into 4.2? >> >> On Mon, Sep 28, 2026 at 10:39 AM huaxin gao <[email protected]> >> wrote: >> >>> +1 >>> >>> I checked: >>> >>> - The tag v4.2.1-rc1 matches the commit in the vote email (72c18760854) >>> and is contained in branch-4.2. >>> - The version is 4.2.1 in both pom.xml and python/pyspark/version.py. >>> - All expected artifacts are present (source, binary distributions, >>> pyspark/pyspark_client/pyspark_connect, SparkR), each with .asc and >>> .sha512. >>> - Verified the SHA512 checksum of spark-4.2.1.tgz: OK. >>> - Verified the GPG signature of spark-4.2.1.tgz: good signature from >>> "Szehon Ho (CODE SIGNING KEY) <[email protected]>". >>> >>> One minor note, not a blocker. The RC is signed with a new key: >>> >>> pub ed25519 2026-09-25 [SC] >>> 9442 A261 0B63 B3F6 1860 285F 43F1 222E 5362 CCF5 >>> uid Szehon Ho (CODE SIGNING KEY) <[email protected]> >>> >>> That key is in dist/dev/spark/KEYS, and the signature verifies correctly >>> against it. However, it is not yet in >>> https://downloads.apache.org/spark/KEYS, >>> which is the file linked in the vote email. Could you append the new key >>> to the >>> release KEYS? >>> >>> Thanks Szehon! >>> >>> On Mon, Sep 28, 2026 at 2:21 AM <[email protected]> wrote: >>> >>>> Please vote on releasing the following candidate as Apache Spark >>>> version 4.2.1. >>>> >>>> The vote is open until Thu, 01 Oct 2026 03:11:20 PDT and passes if a >>>> majority +1 PMC votes are cast, with >>>> a minimum of 3 +1 votes. >>>> >>>> [ ] +1 Release this package as Apache Spark 4.2.1 >>>> [ ] -1 Do not release this package because ... >>>> >>>> To learn more about Apache Spark, please see https://spark.apache.org/ >>>> >>>> The tag to be voted on is v4.2.1-rc1 (commit 72c18760854): >>>> https://github.com/apache/spark/tree/v4.2.1-rc1 >>>> >>>> The release files, including signatures, digests, etc. can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/ >>>> >>>> Signatures used for Spark RCs can be found in this file: >>>> https://downloads.apache.org/spark/KEYS >>>> >>>> The staging repository for this release can be found at: >>>> https://repository.apache.org/content/repositories/orgapachespark-1531/ >>>> >>>> The documentation corresponding to this release can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-docs/ >>>> >>>> The list of bug fixes going into 4.2.1 can be found at the following >>>> URL: >>>> https://issues.apache.org/jira/projects/SPARK/versions/12357191 >>>> >>>> FAQ >>>> >>>> ========================= >>>> How can I help test this release? >>>> ========================= >>>> >>>> If you are a Spark user, you can help us test this release by taking >>>> an existing Spark workload and running on this release candidate, then >>>> reporting any regressions. >>>> >>>> If you're working in PySpark you can set up a virtual env and install >>>> the current RC via "pip install >>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.1-rc1-bin/pyspark-4.2.1.tar.gz >>>> " >>>> and see if anything important breaks. >>>> In the Java/Scala, you can add the staging repository to your project's >>>> resolvers and test >>>> with the RC (make sure to clean up the artifact cache before/after so >>>> you don't end up building with an out of date RC going forward). >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: [email protected] >>>> >>>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >
