Thanks Liang-Chi for raising this. >From the Spark side, if GH-50326 / PR #50327 misses Arrow 25.0.0, Spark likely >cannot pick up the fix until Spark 5.0 around the middle of next year.
Even with Liang-Chi’s Spark-side workaround, the Arrow path is still significantly slower than the existing pickle path, so we would likely need to revert making Arrow serialization the default for regular Python UDFs in Spark 4.2. Spark would like to move to Arrow by default, but we cannot introduce a performance degradation on upgrade, so we hope the community can include GH-50326 / PR #50327 in Arrow 25.0.0. On 2026/07/08 22:31:16 "L. C. Hsieh" wrote: > Hi all, > > I'd like to ask whether the community would consider cancelling RC1 and > cutting an RC2 that includes GH-50326 / PR #50327 (making Array.to_pylist > convert without per-element Scalars; reworked per Antoine's review and > awaiting another pass). > > I realize this doesn't meet the usual bar for sinking an RC — it is a > long-standing inefficiency, not a regression in PyArrow itself. The reason > I'm asking is timing on the Spark side: Spark 4.2, already at RC, switches > regular Python UDFs to Arrow serialization by default, and that path > materializes Arrow columns to Python objects through to_pylist(). With the > current implementation this shows up to Spark users as a performance > regression on upgrade (e.g. to_pylist on a 2M-row list<string> column takes > 1.93s vs 0.34s with the fix; several times slower than the old pickle > path on nested types end-to-end). Arrow 25.0.0 is the last release that can > plausibly reach users while Spark 4.2 rolls out — if the fix slips to > the next Arrow release, early Spark 4.2 adopters would live with this > regression for months. > > A small follow-up is also ready and stacked on it: GH-50429 / PR #50430 > extends the same mechanism to to_pylist(maps_as_pydicts=...), converting > maps to dicts about 21x faster (2.20s → 0.10s for 1M rows) — the > map-to-dict form is exactly what Spark's UDF path consumes. I'd be happy to > have > either just #50327 or both in RC2, whichever the community is comfortable > with. > > If the community prefers to proceed with RC1 as-is, that's completely > understandable — I'll target the next release and we'll carry interim > workarounds on the Spark side. But if there's openness to an RC2, I'll do > my best to get the reviews wrapped up quickly. > > Thanks. > > > On Mon, Jul 6, 2026 at 1:27 AM Raúl Cumplido <[email protected]> wrote: > > > Hi, > > > > I would like to propose the following release candidate (RC1) of Apache > > Arrow version 25.0.0. This is a release consisting of 220 > > resolved GitHub issues[1]. > > > > This release candidate is based on commit: > > 59bea6ec485e7fe351d1aa6753f964f6a6bc353a [2] > > > > The source release rc1 is hosted at [3]. > > The binary artifacts are hosted at [4][5][6][7][8][9]. > > The changelog is located at [10]. > > > > Please download, verify checksums and signatures, run the unit tests, > > and vote on the release. See [11] for the SHA-512 checksum for this RC and > > [12] > > for how to validate a release candidate. > > > > See also a verification result on GitHub pull request [13]. > > > > The vote will be open for at least 72 hours. > > > > [ ] +1 Release this as Apache Arrow 25.0.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache Arrow 25.0.0 because... > > > > [1]: > > https://github.com/apache/arrow/issues?q=is%3Aissue+milestone%3A25.0.0+is%3Aclosed > > [2]: > > https://github.com/apache/arrow/tree/59bea6ec485e7fe351d1aa6753f964f6a6bc353a > > [3]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-25.0.0-rc1 > > [4]: https://packages.apache.org/artifactory/arrow/almalinux-rc/ > > [5]: https://packages.apache.org/artifactory/arrow/amazon-linux-rc/ > > [6]: https://packages.apache.org/artifactory/arrow/centos-rc/ > > [7]: https://packages.apache.org/artifactory/arrow/debian-rc/ > > [8]: https://packages.apache.org/artifactory/arrow/ubuntu-rc/ > > [9]: https://github.com/apache/arrow/releases/tag/apache-arrow-25.0.0-rc1 > > [10]: > > https://github.com/apache/arrow/blob/59bea6ec485e7fe351d1aa6753f964f6a6bc353a/CHANGELOG.md > > [11]: > > 9c7a91eef5177b0339f0479397afcc455f25af4d0cb07cd5ce0dcbdd5bba390ef2310cd114b6769c13f93e7199d58a895b2d680cca997afb6ebdb8e7dabf5b4e > > [12]: https://arrow.apache.org/docs/developers/release_verification.html > > [13]: https://github.com/apache/arrow/pull/50357 > > >
