Hi all,
We are happy to announce the availability of Apache Spark Connect Rust
Client 4.2.0!
This is the first release of the ground-up Rust rewrite of the Spark Connect
Python client (published to PyPI as pyspark-client-rust): a drop-in
replacement for pyspark-client, backed by a native Rust core instead of
py4j/grpcio.
To install the Python client:
pip install pyspark-client-rust==4.2.0
abi3 wheels are published for Linux (x86_64/aarch64), macOS (universal2),
and Windows (x86_64), for CPython 3.9+, along with a source distribution.
Once installed, use it exactly like PySpark.
To use the Rust crates, add to your project's Cargo.toml:
[dependencies]
apache-spark-connect = "4.2.0"
HIGHLIGHTS
- Drop-in PySpark Connect Python API, verified against real PySpark 4.2.0
two ways: the official PySpark test suite runs against this client, and
a
behavioral API-parity sweep compares the Python API surface (functions
and
DataFrame / Column / SparkSession / GroupedData / Window methods)
call-by-call to PySpark.
- Full Spark Connect Rust API coverage, including Rust UDFs.
- Native Rust transport via tonic; cloudpickle vendored to match upstream,
so there is no grpcio/py4j at runtime, plus native type conversion.
WHY DOES THE VERSION START AT 4.2.0?
Even though this is the first release of the rewrite, the version starts at
4.2.0 to track the Apache Spark / PySpark version it targets. This (1)
removes
any ambiguity about which Spark version a given client is compatible with,
and
(2) lets PySpark users pip install pyspark-client-rust==<version> exactly as
they would pyspark-client, keeping the same versioning across both.
DOCUMENTATION
https://apache.github.io/spark-connect-rust/
We would like to thank all contributors who made this release possible.
Thanks,
Hyukjin Kwon
