I also unfortunately encountered a bug in the spark-pipelines CLI that causes it to fail when the spec file is in a different directory than the CWD. I'm not sure whether this merits a new RC, but posting here in case.
JIRA: https://issues.apache.org/jira/browse/SPARK-54508 PR: https://github.com/apache/spark/pull/53219 On Tue, Nov 25, 2025 at 9:47 AM Sandy Ryza <[email protected]> wrote: > One thing I did just notice is that the pyspark.pipelines package is not > included in the API reference docs. Are pure docs site changes part of the > release, or is there an ability to push them independently? > > PR to fix: https://github.com/apache/spark/pull/53213 > > On Tue, Nov 25, 2025 at 9:12 AM Kumar Chinnakali < > [email protected]> wrote: > >> [email protected] >> >> On Tue, Nov 25, 2025 at 11:05 AM Sandy Ryza <[email protected]> wrote: >> >>> +1 (non-binding) >>> >>> Installed via `pip install >>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz` >>> and verified functionality of pyspark and spark-pipelines. >>> >>> On Sun, Nov 23, 2025 at 7:40 PM <[email protected]> wrote: >>> >>>> Please vote on releasing the following candidate as Apache Spark >>>> version 4.1.0. >>>> >>>> The vote is open until Wed, 26 Nov 2025 20:40:01 PST and passes if a >>>> majority +1 PMC votes are cast, with >>>> a minimum of 3 +1 votes. >>>> >>>> [ ] +1 Release this package as Apache Spark 4.1.0 >>>> [ ] -1 Do not release this package because ... >>>> >>>> To learn more about Apache Spark, please see https://spark.apache.org/ >>>> >>>> The tag to be voted on is v4.1.0-rc1 (commit 2808b96a1b8): >>>> https://github.com/apache/spark/tree/v4.1.0-rc1 >>>> >>>> The release files, including signatures, digests, etc. can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/ >>>> >>>> Signatures used for Spark RCs can be found in this file: >>>> https://downloads.apache.org/spark/KEYS >>>> >>>> The staging repository for this release can be found at: >>>> https://repository.apache.org/content/repositories/orgapachespark-1506/ >>>> >>>> The documentation corresponding to this release can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-docs/ >>>> >>>> The list of bug fixes going into 4.1.0 can be found at the following >>>> URL: >>>> https://issues.apache.org/jira/projects/SPARK/versions/12355581 >>>> >>>> FAQ >>>> >>>> ========================= >>>> How can I help test this release? >>>> ========================= >>>> >>>> If you are a Spark user, you can help us test this release by taking >>>> an existing Spark workload and running on this release candidate, then >>>> reporting any regressions. >>>> >>>> If you're working in PySpark you can set up a virtual env and install >>>> the current RC via "pip install >>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz >>>> " >>>> and see if anything important breaks. >>>> In the Java/Scala, you can add the staging repository to your project's >>>> resolvers and test >>>> with the RC (make sure to clean up the artifact cache before/after so >>>> you don't end up building with an out of date RC going forward). >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: [email protected] >>>> >>>>
