L. C. Hsieh created SPARK-58050:
-----------------------------------
Summary: Fuse per-row converters into bulk Arrow-to-rows
conversion and bulk-assemble Arrow Python UDF results
Key: SPARK-58050
URL: https://issues.apache.org/jira/browse/SPARK-58050
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-58019/SPARK-58023/SPARK-58024. Even with bulk
{{_to_pylist}}, the Arrow Python UDF worker still runs per-row Python
converters on both sides: on input, {{_create_converter}} wraps every map row
into a dict and every struct row into a Row one value at a time; on output,
{{LocalDataToArrowConversion}} converters copy every returned list, convert
every dict to an entry list and every Row to a dict before {{pa.array}}. Worker
profiling shows these per-row converters dominate the remaining gap vs pickled
UDFs on nested types (pickle currently beats arrow by 1.4-3.1x on
array/map/struct).
This change:
* adds {{ArrowTableToRowsConversion._to_rows_column}}, fusing the per-row input
converter into the bulk conversion (map rows become dicts and struct rows
become Rows built from flattened child columns; child converters are applied
per flattened column),
* bulk-assembles UDF results when element/field/value converters are identity:
arrays pass the returned lists to {{pa.array}} directly, map results pass dicts
directly (PyArrow accepts dicts for map types), struct results are transposed
and assembled via {{StructArray.from_arrays}}; anything else falls back to the
existing per-row path.
Microbenchmark (400k rows): input map->dict 3.0x, input struct->Row 1.8x,
output array 7.5x, output map 4.2x, output struct 4.0x — outputs identical.
End-to-end UDF benchmark (6.4M rows): array_string 2.39s -> 1.77s,
map<string,int> 3.65s -> 2.40s, struct 6.03s -> 4.87s on top of
SPARK-58019/58023/58024; arrow-vs-pickle outputs verified identical with
injected nulls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
