[
https://issues.apache.org/jira/browse/SPARK-61181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61181:
-----------------------------------
Labels: pull-request-available (was: )
> Read the local relation settings of the server for local relations of 64 KiB
> or more, and raise LOCAL_RELATION_SIZE_LIMIT_EXCEEDED, in the Rust and Python
> clients as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61181
> URL: https://issues.apache.org/jira/browse/SPARK-61181
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's createDataFrame reads
> spark.sql.session.localRelationCacheThreshold and
> spark.sql.session.localRelationSizeLimit for every local relation: at the
> threshold it uploads the relation as cache artifacts, and past the size limit
> it raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
> The Rust client reads them only for a relation of at least 32 MiB of Arrow
> data, and sends a smaller one inline. So, with both settings lowered, local
> rows and some pandas frames (those of the float64 and string fast paths)
> under 32 MiB are sent without the error, while the same frame with a DDL
> schema goes through the Arrow importer and raises it. For example, with a
> threshold of 1 byte and a size limit of 100 bytes, createDataFrame of 2,000
> rows of a long and a 100-character string (about 225 KB) succeeds, where
> PySpark raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED. With the default settings
> too, a relation of 1 MiB (the default threshold) to 32 MiB is sent inline,
> where PySpark uploads it as cache artifacts.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
