James Willis created SPARK-59432:
------------------------------------
Summary: Support free-threaded (no-GIL) CPython in
pyspark-client-rust
Key: SPARK-59432
URL: https://issues.apache.org/jira/browse/SPARK-59432
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: connect-rust-4.2.0
Reporter: James Willis
h2. Background
{{pyspark-client-rust}} is a PyO3 extension. On free-threaded CPython it does
not currently work GIL-off, for two independent reasons:
# The {{#[pymodule]}} in {{crates/pyspark-rs/src/lib.rs}} does not declare
{{{}gil_used = false{}}}. CPython re-enables the GIL process-wide when it
imports an extension module that has not opted in, so even on a free-threaded
interpreter the GIL is silently switched back on.
# The published wheels are {{{}cp39-abi3{}}}. The stable ABI (abi3) and the
free-threaded ABI are mutually exclusive, so no wheel is selectable on a
free-threaded interpreter at all.
h2. Why this matters
A pure-Rust (tonic) Spark Connect client is uniquely able to run with the GIL
disabled: unlike the classic PySpark client, it pulls in no {{{}grpcio{}}},
which otherwise re-enables the GIL on import (grpc/grpc#38762). Free-threading
lets a Python application drive Spark Connect with real thread parallelism on
the client side.
h2. Proposed change
# Declare {{#[pymodule(gil_used = false)]}} on {{_pyspark
}}({{{}crates/pyspark-rs/src/lib.rs{}}}). This is inert on GIL-enabled builds,
so it is safe to ship on its own.
# Add a free-threaded wheel build leg (cp313t / cp314t) to
{{{}release.yml{}}}, in addition to the existing abi3 leg.
# Add a smoke test that imports {{pyspark_client_rust}} on a free-threaded
interpreter and asserts {{{}sys._is_gil_enabled() is False{}}}.
h2. Safety
The crate is already thread-safe, so {{gil_used = false}} is a truthful
assertion:
* The global tokio runtime is a {{OnceLock<Runtime>}} (multi-threaded), and
every blocking call is wrapped in {{py.detach(|| block_on(...))}} (GIL released
while blocking).
* All process-global state is behind {{OnceLock}} / {{Mutex}} / atomics (e.g.
the active-session {{{}OnceLock<Mutex<Option<SparkSession>>>{}}}).
* Every {{#[pyclass]}} is {{Sync}} (PyO3's free-threaded requirement, enforced
at compile time): the core {{SparkSession}} is built from {{Arc}} /
{{Arc<Mutex<..>> }}/ atomics; stored {{Py<T>}} handles are {{{}Send + Sync{}}};
the response stream's {{tonic::Streaming}} is made {{Sync}} via
{{{}sync_wrapper::SyncWrapper{}}}.
* No {{Rc}} / {{RefCell}} / {{Cell}} / {{static mut}} anywhere in the
extension.
Behavioral note (not a soundness issue): under free-threading, two Python
threads invoking a {{&mut self}} method on the *same* object instance get a
{{RuntimeError: Already borrowed}} rather than corrupting state. That surface
is small (consume-once builders, {{{}ResponseStream::_{_}next{_}_{}}}) and this
is the documented free-threaded contract.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
