[
https://issues.apache.org/jira/browse/SPARK-58019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-58019:
-----------------------------------
Assignee: L. C. Hsieh
> Convert Arrow list columns to Python rows in bulk
> -------------------------------------------------
>
> Key: SPARK-58019
> URL: https://issues.apache.org/jira/browse/SPARK-58019
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> PyArrow's {{Array.to_pylist()}} materializes one Scalar per element; for
> list-typed columns each row additionally allocates a C++ scalar, a Python
> Scalar wrapper and a Python Array wrapper before converting elements one by
> one. This makes Arrow-optimized Python UDF inputs and Spark Connect
> {{collect()}} several times slower on array columns than necessary. This is
> the root cause of the performance regression observed when Arrow-serialized
> Python UDFs read array columns.
> Upstream Arrow issue: https://github.com/apache/arrow/issues/50326 (fix
> proposed in https://github.com/apache/arrow/pull/50327), but that will only
> be available in a future PyArrow release.
> Until the minimum supported PyArrow version includes the fix, PySpark can
> convert list columns in bulk itself using only public PyArrow APIs: convert
> the flattened child values of a list column in a single pass, then slice the
> resulting Python list per row using the offsets and validity bitmap. Leaf
> values are still converted by Arrow's own {{to_pylist}}, so results are
> exactly identical (unlike a pandas round trip, which coerces {{[1, None, 3]}}
> in an int list to floats/NaN).
> ASV microbenchmark (1M rows): {{list<string>}} 769ms -> 507ms (1.5x);
> {{list<list<int32>>}} with nulls 1.86s -> 537ms (3.5x); peak memory unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
