+1 On Thu, Sep 3, 2026 at 6:07 AM L. C. Hsieh <[email protected]> wrote:
> +1, on the technical merits. > > I verified RC2 end-to-end and the software is in good shape — build, > tests, and > functional results are all correct (details below). Two > packaging/release-policy > items stood out that I'd ask the RM to look at before this is finalized; I > don't > think either is a code problem, but the first in particular may matter for > a > formal ASF release, so I'm noting them rather than blocking. > > ## What I verified (all passed) > > Identity & integrity: > - git tag v4.2.0-rc2 resolves to commit ce1fa08 (matches the vote email). > - Downloaded macOS universal2 wheel and the sdist; both SHA-256 match the > digests GitHub recorded for the assets. > - proto pinning in the tree is PROTO_VERSION=v4.2.0 / SPARK_SHA=32f7299, > consistent with targeting Spark 4.2.0. > > Python wheel (installed into a throwaway venv): > - The macOS universal2 wheel installs and imports as a drop-in `pyspark` > (pyspark-client-rust 4.2.0). The abi3 native extension is a genuine > universal2 binary (x86_64 + arm64 slices) and loads on Apple Silicon. > - Ran real workloads against a local Spark 4.2.0 Connect server (the > bundled > spark-connect_2.13-4.2.0.jar from the apache/spark:4.2.0 image): > range/count, > groupBy aggregation, SQL, createDataFrame + filter — all correct; > server-reported spark.version = 4.2.0. > - collect() and toPandas() type conversion is correct across > int/string/double/ > boolean and NULL->NaN. > > Rust crates (from the tag): > - cargo build --release and cargo test --release both pass, no failures > (server-dependent e2e tests are ignored by design without a live server). > - The Rust `sql` example runs end-to-end against the live server (SQL -> > write > parquet -> read back), confirming the native tonic transport works > against a > real Spark 4.2.0 server. > > ## Notes for the RM > > 1. The candidate is not published to the ASF distribution area and is not > signed. The release files live only on GitHub Releases. There > is no .asc PGP signature, no .sha512, and no reference to the Spark KEYS > file. GitHub's SHA-256 digests establish transfer integrity only, not > that > a release manager signed the artifacts. ASF policy is that the thing > being > voted on is a signed source release in the dist area; convenience > binaries > are not the subject of the vote. Notably the sibling Spark Connect Go > client > RC does publish to dist.apache.org, so there is an established pattern > here > for this candidate to follow. > > 2. The source distribution does not ship a LICENSE. The sdist > (pyspark_client_rust-4.2.0.tar.gz) has 390 files and none of them is a > LICENSE file, so downstream repackagers get no license text — this one > should be fixed regardless. Separately, the tree bundles some vendored > third-party code, so a NOTICE also looks like it's needed here (and > there > isn't one); worth the RM double-checking the LICENSE/NOTICE against > what's > actually bundled. > > ## Minor (non-blocking) observation > > DataFrame.show() renders rows in a compact form (e.g. `[apple, 123]`) > rather > than PySpark's bordered ASCII table. Values are correct; flagging only > because > the client is presented as a strict drop-in and some users scrape show() > output. > > > Thanks for driving this — the technical work looks solid. > > > On Tue, Sep 1, 2026 at 3:08 PM Hyukjin Kwon <[email protected]> wrote: > >> Please vote on releasing the following candidate as Apache Spark Connect >> Rust >> Client 4.2.0 RC2. >> >> This is the *first* release of the ground-up Rust rewrite of the Spark >> Connect >> Python client (published to PyPI as pyspark-client-rust): a drop-in >> replacement >> for pyspark-client <https://pypi.org/project/pyspark-client/>, backed by >> a native Rust core instead of py4j/grpcio. >> >> The vote is open until Thursday, September 7, 2026, 5:00 PM PDT and >> passes if a majority of +1 PMC votes are cast, with a minimum of >> 3 +1 votes. >> >> [ ] +1 Release this package as Apache Spark Connect Rust Client 4.2.0 >> [ ] -1 Do not release this package because … >> >> TAG: >> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2 >> (commit: ce1fa08) >> >> RELEASE FILES (wheels + sdist, built and attached by GitHub Actions): >> https://github.com/apache/spark-connect-rust/releases/tag/v4.2.0-rc2 >> >> DOCUMENTATION: >> https://apache.github.io/spark-connect-rust/ >> >> LIST OF ISSUES: >> https://issues.apache.org/jira/projects/SPARK/versions/12357484 >> HOW TO TRY IT — PYTHON (pip) >> >> The release attaches abi3 wheels (Linux x86_64/aarch64, macOS universal2, >> Windows x86_64; CPython 3.9+) plus a source distribution. Grab the wheel >> for >> your platform from the release page and install it directly, e.g.: >> >> pip install >> https://github.com/apache/spark-connect-rust/releases/download/v4.2.0-rc2/pyspark_client_rust-4.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl >> >> Then use it exactly like PySpark. See if your existing workloads work >> with this. >> HOW TO TEST IT — RUST (cargo) >> >> The Rust crates are intentionally NOT on crates.io for a release >> candidate >> (crates.io is immutable; the crates are published there only when the >> release is >> finalized). Test against the tagged source instead: >> >> # A) Pin a git dependency to the RC tag in your project's Cargo.toml: >> [dependencies] >> apache-spark-connect = { git = >> "https://github.com/apache/spark-connect-rust";, tag = "v4.2.0-rc2" } >> >> # B) Or build and run the crates' own tests from the tag: >> git clone --branch v4.2.0-rc2 https://github.com/apache/spark-connect-rust >> cd spark-connect-rust >> cargo build --release >> cargo test >> >> HIGHLIGHTS >> >> - *Drop-in PySpark Connect Python API*, verified against real PySpark >> 4.2.0 two >> ways: the official PySpark test suite runs against this client, and a >> behavioral API-parity sweep compares the Python API surface >> (functions and >> DataFrame / Column / SparkSession / GroupedData / Window methods) >> call-by-call >> to PySpark. >> - Spark Connect Rust API *full* coverage, including *Rust UDF*s >> (docs: https://apache.github.io/spark-connect-rust/). >> - *Native Rust transport* via tonic; cloudpickle vendored to match >> upstream, so >> no grpcio/py4j at runtime. and *native type conversion.* >> >> WHY DOES THE VERSION START AT 4.2.0? >> >> Even though this is the first release of the rewrite, the version starts >> at >> 4.2.0 to track the Apache Spark / PySpark version it targets. This (1) >> removes >> any ambiguity about which Spark version a given client is compatible >> with, and >> (2) lets PySpark users pip install pyspark-client-rust== exactly >> as they would pyspark-client, keeping the same versioning across both. >> >> Thanks, >> Hyukjin Kwon >> > -- Ruifeng Zheng E-mail: [email protected]
