[
https://issues.apache.org/jira/browse/SPARK-59110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59110.
----------------------------------
Fix Version/s: connect-rust-4.2.0
connect-rust-4.3.0
Resolution: Fixed
> Fix PySpark functions parity in the Rust drop-in: ColumnOrName string args,
> agg dict form, variadic concat_ws/format_string
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59110
> URL: https://issues.apache.org/jira/browse/SPARK-59110
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.2.0, connect-rust-4.3.0
>
>
> A behavioral parity sweep of the Rust drop-in against real PySpark 4.2.0
> (functions called with string column names + a 51-op method sweep) found
> systemic gaps the transport-level suite never exercised.
> 1. functions ColumnOrName parity: the shared to_column coercion mapped a
> Python str to a string LITERAL, but PySpark treats a bare str as a COLUMN
> NAME. ~130 functions were wrong (e.g. sqrt("v") errored; upper("v"),
> struct("id","g") silently used literals). Fixed: _create_wrapper resolves
> column-position str args via col(), driven by a _FN_COL_ARGS map derived from
> real PySpark 4.2.0 signatures; lit() and true literal params (date_format
> format, regexp_replace pattern, etc.) stay literals.
> 2. GroupedData.agg dict form agg({"col":"func"}) was unsupported; now
> accepted.
> 3. Variadic concat_ws(sep, *cols) and format_string(format, *cols) dropped
> extra columns (core took a single Column, dispatch forwarded only args[0]);
> made variadic like concat.
> Verified against a live Spark Connect server: functions string==Column sweep
> 0 divergences (was 73 errors + ~50 wrong), 51-op method sweep 51/51, offline
> suite 232 passed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
