Hyukjin Kwon created SPARK-59115:
------------------------------------
Summary: Complete PySpark 4.2.0 behavioral parity for the Rust
drop-in (window functions, chained when, lag/lead, json_tuple, parameterized
SQL, struct->Row, na.fill, nested/empty createDataFrame)
Key: SPARK-59115
URL: https://issues.apache.org/jira/browse/SPARK-59115
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.2.0
Reporter: Hyukjin Kwon
Complete PySpark 4.2.0 behavioral parity for the Rust drop-in, driving an
exhaustive differential harness (real PySpark 4.2.0 Connect client vs the
drop-in, same ops on the same server) to zero: 143/143 identical. The native
Rust engine (crates/spark-connect) was already correct; these were
binding-layer (crates/pyspark-rs + python/pyspark) gaps plus a few core
function-signature extensions.
Fixes:
- na.fill/fillna accept float/string/bool and the {col: value} dict form (were
int-only); single-field DDL schema strings parse as a schema; createDataFrame
accepts empty data with an explicit schema.
- Window subsystem: Window.partitionBy/orderBy coerced a bare string to a
literal (windows partitioned/ordered by a constant), and the ROWS frame
boundary had the wrong type ->
row_number/rank/dense_rank/cume_dist/percent_rank/lag/lead/nth_value and sum
over rows/rangeBetween now correct.
- Chained when().when().otherwise() (the free function built an
UnresolvedFunction instead of a CaseWhen, dropping earlier branches).
- lag/lead accept offset/default; json_tuple is variadic over literal field
names.
- Parameterized spark.sql(query, args=[...] | {...});
struct/named_struct/from_json return a Row (not a dict) for struct-typed
values; createDataFrame with array<struct> data.
Verified: differential harness 143/143 (zero divergences), three behavioral
batteries 32/32 + 83/83 + 56/56, offline suite 236 passed, cargo fmt/build
clean, ruff clean.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
