[
https://issues.apache.org/jira/browse/SPARK-58023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095070#comment-18095070
]
L. C. Hsieh commented on SPARK-58023:
-------------------------------------
Closed in favor of the upstream fix: apache/arrow#50327 (GH-50326) implements
the leaf conversion natively in PyArrow (planned for 25.0.1), and SPARK-58019
gates on the installed PyArrow version so Spark picks it up automatically. See
https://github.com/apache/spark/pull/57104#issuecomment for the full rationale.
> Fast paths for converting Arrow string, binary, numeric and boolean columns
> to Python rows
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-58023
> URL: https://issues.apache.org/jira/browse/SPARK-58023
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Follow-up of SPARK-58019. {{ArrowTableToRowsConversion._to_pylist}} still
> converts leaf (non-nested) values one Scalar at a time via {{to_pylist()}}.
> Add fast paths that keep exact {{as_py}} semantics:
> * string/large_string/binary/large_binary: Arrow's object-dtype NumPy
> conversion produces exactly {{str}}/{{bytes}}/{{None}}.
> * integer/float32/float64/boolean: zero-copy NumPy view + {{ndarray.tolist}};
> with nulls, fill with a placeholder ({{pc.fill_null}}) and restore {{None}}
> from the validity bitmap, so ints are materialized from the int buffer and
> never coerced through floats.
> Types whose {{as_py}} returns non-primitive objects (dates, timestamps,
> decimals, ...) keep using {{to_pylist}}.
> ASV microbenchmark (1M rows): string with nulls 196ms -> 20ms (9.7x); int64
> with nulls 99ms -> 28ms (3.6x); float64 without nulls 100ms -> 9ms (11x).
> End-to-end {{list<string>}} conversion improves from 507ms to 118ms on top of
> SPARK-58019.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
