Hyukjin Kwon created SPARK-59110:
------------------------------------
Summary: Fix PySpark functions parity in the Rust drop-in:
ColumnOrName string args, agg dict form, variadic concat_ws/format_string
Key: SPARK-59110
URL: https://issues.apache.org/jira/browse/SPARK-59110
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.2.0
Reporter: Hyukjin Kwon
A behavioral parity sweep of the Rust drop-in against real PySpark 4.2.0
(functions called with string column names + a 51-op method sweep) found
systemic gaps the transport-level suite never exercised.
1. functions ColumnOrName parity: the shared to_column coercion mapped a Python
str to a string LITERAL, but PySpark treats a bare str as a COLUMN NAME. ~130
functions were wrong (e.g. sqrt("v") errored; upper("v"), struct("id","g")
silently used literals). Fixed: _create_wrapper resolves column-position str
args via col(), driven by a _FN_COL_ARGS map derived from real PySpark 4.2.0
signatures; lit() and true literal params (date_format format, regexp_replace
pattern, etc.) stay literals.
2. GroupedData.agg dict form agg({"col":"func"}) was unsupported; now accepted.
3. Variadic concat_ws(sep, *cols) and format_string(format, *cols) dropped
extra columns (core took a single Column, dispatch forwarded only args[0]);
made variadic like concat.
Verified against a live Spark Connect server: functions string==Column sweep 0
divergences (was 73 errors + ~50 wrong), 51-op method sweep 51/51, offline
suite 232 passed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
