[
https://issues.apache.org/jira/browse/SPARK-61145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61145.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 334
[https://github.com/apache/spark-connect-rust/pull/334]
> Calls from several Python threads on one DataFrameWriter, DataFrameWriterV2
> or DataStreamWriter fail with "Already borrowed"
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61145
> URL: https://issues.apache.org/jira/browse/SPARK-61145
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The writer methods of the Python client that release the GIL while the
> request runs (save, parquet, csv, saveAsTable, create, append, start,
> toTable, ...) took an exclusive Rust borrow of the writer object for the
> whole request. A second thread that called a method of the same writer, a
> write or a builder such as format or mode, got RuntimeError: Already
> borrowed. PySpark's writers are plain Python objects, so these calls run and
> each write goes through. The writer state is now behind a lock that is held
> only to copy or replace it, so that a request runs on its own copy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
