I am also evaluating / testing the design of SupportsPushDownVariants in downstream connectors. There is a chance we may either patch the design in Spark or roll that feature back from 4.1 to give us a bit more time. Will report back tomorrow.
- Anton вт, 25 лист. 2025 р. о 13:08 Szehon Ho <[email protected]> пише: > Hi, I think we need to fix a simple bug in a new metric in numSourceRows, > introduced in this pr: https://github.com/apache/spark/pull/52669. The > author will make one soon. > > Also as you know, there is a bug being fixed now that makes Merge Into > with schema evolution not work with Dataframe API: > https://github.com/apache/spark/pull/53207 > > Thanks, > Szehon > > On Tue, Nov 25, 2025 at 11:18 AM Sandy Ryza <[email protected]> wrote: > >> I also unfortunately encountered a bug in the spark-pipelines CLI >> that causes it to fail when the spec file is in a different directory than >> the CWD. I'm not sure whether this merits a new RC, but posting here in >> case. >> >> JIRA: https://issues.apache.org/jira/browse/SPARK-54508 >> PR: https://github.com/apache/spark/pull/53219 >> >> On Tue, Nov 25, 2025 at 9:47 AM Sandy Ryza <[email protected]> wrote: >> >>> One thing I did just notice is that the pyspark.pipelines package is not >>> included in the API reference docs. Are pure docs site changes part of the >>> release, or is there an ability to push them independently? >>> >>> PR to fix: https://github.com/apache/spark/pull/53213 >>> >>> On Tue, Nov 25, 2025 at 9:12 AM Kumar Chinnakali < >>> [email protected]> wrote: >>> >>>> [email protected] >>>> >>>> On Tue, Nov 25, 2025 at 11:05 AM Sandy Ryza <[email protected]> >>>> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> Installed via `pip install >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz` >>>>> and verified functionality of pyspark and spark-pipelines. >>>>> >>>>> On Sun, Nov 23, 2025 at 7:40 PM <[email protected]> wrote: >>>>> >>>>>> Please vote on releasing the following candidate as Apache Spark >>>>>> version 4.1.0. >>>>>> >>>>>> The vote is open until Wed, 26 Nov 2025 20:40:01 PST and passes if a >>>>>> majority +1 PMC votes are cast, with >>>>>> a minimum of 3 +1 votes. >>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 4.1.0 >>>>>> [ ] -1 Do not release this package because ... >>>>>> >>>>>> To learn more about Apache Spark, please see >>>>>> https://spark.apache.org/ >>>>>> >>>>>> The tag to be voted on is v4.1.0-rc1 (commit 2808b96a1b8): >>>>>> https://github.com/apache/spark/tree/v4.1.0-rc1 >>>>>> >>>>>> The release files, including signatures, digests, etc. can be found >>>>>> at: >>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/ >>>>>> >>>>>> Signatures used for Spark RCs can be found in this file: >>>>>> https://downloads.apache.org/spark/KEYS >>>>>> >>>>>> The staging repository for this release can be found at: >>>>>> >>>>>> https://repository.apache.org/content/repositories/orgapachespark-1506/ >>>>>> >>>>>> The documentation corresponding to this release can be found at: >>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-docs/ >>>>>> >>>>>> The list of bug fixes going into 4.1.0 can be found at the following >>>>>> URL: >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12355581 >>>>>> >>>>>> FAQ >>>>>> >>>>>> ========================= >>>>>> How can I help test this release? >>>>>> ========================= >>>>>> >>>>>> If you are a Spark user, you can help us test this release by taking >>>>>> an existing Spark workload and running on this release candidate, then >>>>>> reporting any regressions. >>>>>> >>>>>> If you're working in PySpark you can set up a virtual env and install >>>>>> the current RC via "pip install >>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz >>>>>> " >>>>>> and see if anything important breaks. >>>>>> In the Java/Scala, you can add the staging repository to your >>>>>> project's resolvers and test >>>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>>> you don't end up building with an out of date RC going forward). >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: [email protected] >>>>>> >>>>>>
