[
https://issues.apache.org/jira/browse/SPARK-59432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59432:
-----------------------------------
Labels: pull-request-available (was: )
> Support free-threaded (no-GIL) CPython in pyspark-client-rust
> -------------------------------------------------------------
>
> Key: SPARK-59432
> URL: https://issues.apache.org/jira/browse/SPARK-59432
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: James Willis
> Priority: Minor
> Labels: pull-request-available
>
> h2. Background
> {{pyspark-client-rust}} is a PyO3 extension. On free-threaded CPython it does
> not currently work GIL-off, for two independent reasons:
> # The {{#[pymodule]}} in {{crates/pyspark-rs/src/lib.rs}} does not declare
> {{{}gil_used = false{}}}. CPython re-enables the GIL process-wide when it
> imports an extension module that has not opted in, so even on a free-threaded
> interpreter the GIL is silently switched back on.
> # The published wheels are {{{}cp39-abi3{}}}. The stable ABI (abi3) and the
> free-threaded ABI are mutually exclusive, so no wheel is selectable on a
> free-threaded interpreter at all.
> h2. Why this matters
> A pure-Rust (tonic) Spark Connect client is uniquely able to run with the GIL
> disabled: unlike the classic PySpark client, it pulls in no {{{}grpcio{}}},
> which otherwise re-enables the GIL on import (grpc/grpc#38762).
> Free-threading lets a Python application drive Spark Connect with real thread
> parallelism on the client side.
> h2. Proposed change
> # Declare {{#[pymodule(gil_used = false)]}} on {{_pyspark
> }}({{{}crates/pyspark-rs/src/lib.rs{}}}). This is inert on GIL-enabled
> builds, so it is safe to ship on its own.
> # Add a free-threaded wheel build leg (cp313t / cp314t) to
> {{{}release.yml{}}}, in addition to the existing abi3 leg.
> # Add a smoke test that imports {{pyspark_client_rust}} on a free-threaded
> interpreter and asserts {{{}sys._is_gil_enabled() is False{}}}.
> h2. Safety
> The crate is already thread-safe, so {{gil_used = false}} is a truthful
> assertion:
> * The global tokio runtime is a {{OnceLock<Runtime>}} (multi-threaded), and
> every blocking call is wrapped in {{py.detach(|| block_on(...))}} (GIL
> released while blocking).
> * All process-global state is behind {{OnceLock}} / {{Mutex}} / atomics
> (e.g. the active-session {{{}OnceLock<Mutex<Option<SparkSession>>>{}}}).
> * Every {{#[pyclass]}} is {{Sync}} (PyO3's free-threaded requirement,
> enforced at compile time): the core {{SparkSession}} is built from {{Arc}} /
> {{Arc<Mutex<..>> }}/ atomics; stored {{Py<T>}} handles are {{{}Send +
> Sync{}}}; the response stream's {{tonic::Streaming}} is made {{Sync}} via
> {{{}sync_wrapper::SyncWrapper{}}}.
> * No {{Rc}} / {{RefCell}} / {{Cell}} / {{static mut}} anywhere in the
> extension.
> Behavioral note (not a soundness issue): under free-threading, two Python
> threads invoking a {{&mut self}} method on the *same* object instance get a
> {{RuntimeError: Already borrowed}} rather than corrupting state. That surface
> is small (consume-once builders, {{{}ResponseStream::_{_}next{_}_{}}}) and
> this is the documented free-threaded contract.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
