Hyukjin Kwon created SPARK-59113:
------------------------------------
Summary: Fix PySpark UDF parity in the Rust drop-in: string args,
pandas_udf family, applyInPandas/mapInPandas/mapInArrow/cogroup, and
udf.register
Key: SPARK-59113
URL: https://issues.apache.org/jira/browse/SPARK-59113
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.2.0
Reporter: Hyukjin Kwon
A full behavioral UDF sweep of the Rust drop-in against real PySpark 4.2.0
(live Spark Connect server) found the UDF subsystem broadly broken; this fixes
it end to end (battery 32/32, offline suite 236 passed).
- Scalar/arrow-optimized python UDFs treated a string column arg as a literal
(udf(f)("c") vs udf(f)(F.col("c"))); and @udf/@pandas_udf/@arrow_udf decorator
forms raised TypeError. Fixed the UDF __call__ arg coercion (str->col) and the
decorator/return-type handling.
- pandas_udf family: the command pickled a DDL string instead of a concrete
DataType (worker Arrow serializer failed), and eval-type wasn't inferred from
type hints. Now pickles a real DataType and infers eval type (scalar 200 /
iterator 204 / grouped-agg 202).
- Grouped/map UDFs (applyInPandas, mapInPandas, mapInArrow, applyInArrow,
cogroup): the GroupMap/MapPartitions/CoGroupMap relation didn't pass input
column names, so the worker DataFrame was unnamed (KeyError). Now passes the
child columns as the UDF arguments.
- spark.udf.register(name, f, T) didn't notify the server (UNRESOLVED_ROUTINE
in SQL). Now sends a RegisterFunction command with the cloudpickled PythonUDF.
Verified live: pandas_udf scalar/iterator/grouped-agg,
applyInPandas/mapInPandas/mapInArrow/applyInArrow/cogroup, register+SQL all
return correct results.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
