[
https://issues.apache.org/jira/browse/SPARK-61143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61143:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Decode the Arrow batches that the server compresses with zstd or lz4 in the
> Rust client, as PySpark does
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61143
> URL: https://issues.apache.org/jira/browse/SPARK-61143
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> With spark.sql.execution.arrow.compression.codec set to zstd or lz4, the
> Spark Connect server compresses the Arrow batches of the results. PySpark
> reads them with PyArrow.
> The Rust client, and so the Python client, builds arrow without the
> ipc_compression feature, and fails to decode them:
> PySparkRuntimeError: Failed to decode Arrow batch: Invalid argument error:
> zstd IPC decompression requires the zstd feature
> So collect, toArrow and toPandas fail, and so do the results of pandas UDFs
> and applyInPandas under that conf. Nine tests of Apache Spark's
> test_parity_arrow.py, test_parity_pandas_grouped_map.py and
> test_parity_pandas_udf_scalar.py (*_with_compression_codec*) fail on this
> client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
