[
https://issues.apache.org/jira/browse/SPARK-61113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61113.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 310
[https://github.com/apache/spark-connect-rust/pull/310]
> Add DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the
> Python client as PySpark does
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61113
> URL: https://issues.apache.org/jira/browse/SPARK-61113
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark 4.3.0's DataFrame implements the stream of the Arrow PyCapsule
> interface. {{DataFrame.__arrow_c_stream__(requested_schema=None)}} exports
> the rows of the DataFrame as an Arrow C stream
> ({{pyspark.sql.interchange.SparkArrowCStreamer}}). It reads one partition at
> a time, through {{mapInArrow}} and {{toLocalIterator}}, so the client never
> holds all the rows at once. Libraries that read Arrow streams then take a
> Spark DataFrame directly, for example
> {{pa.RecordBatchReader.from_stream(df)}}, Polars or DuckDB.
> The Python client has neither {{DataFrame.__arrow_c_stream__}} nor
> {{pyspark.sql.interchange}}. {{pa.RecordBatchReader.from_stream(df)}} raises
> {{TypeError: Expected an object implementing the Arrow PyCapsule Protocol for
> streams}}.
> The fix is to vendor {{pyspark.sql.interchange}} from Apache Spark v4.3.0 and
> to add {{__arrow_c_stream__}} to the DataFrame, as PySpark has them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
