[
https://issues.apache.org/jira/browse/SPARK-59113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59113.
----------------------------------
Fix Version/s: connect-rust-4.2.0
connect-rust-4.3.0
Resolution: Fixed
> Fix PySpark UDF parity in the Rust drop-in: string args, pandas_udf family,
> applyInPandas/mapInPandas/mapInArrow/cogroup, and udf.register
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59113
> URL: https://issues.apache.org/jira/browse/SPARK-59113
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.2.0, connect-rust-4.3.0
>
>
> A full behavioral UDF sweep of the Rust drop-in against real PySpark 4.2.0
> (live Spark Connect server) found the UDF subsystem broadly broken; this
> fixes it end to end (battery 32/32, offline suite 236 passed).
> - Scalar/arrow-optimized python UDFs treated a string column arg as a literal
> (udf(f)("c") vs udf(f)(F.col("c"))); and @udf/@pandas_udf/@arrow_udf
> decorator forms raised TypeError. Fixed the UDF __call__ arg coercion
> (str->col) and the decorator/return-type handling.
> - pandas_udf family: the command pickled a DDL string instead of a concrete
> DataType (worker Arrow serializer failed), and eval-type wasn't inferred from
> type hints. Now pickles a real DataType and infers eval type (scalar 200 /
> iterator 204 / grouped-agg 202).
> - Grouped/map UDFs (applyInPandas, mapInPandas, mapInArrow, applyInArrow,
> cogroup): the GroupMap/MapPartitions/CoGroupMap relation didn't pass input
> column names, so the worker DataFrame was unnamed (KeyError). Now passes the
> child columns as the UDF arguments.
> - spark.udf.register(name, f, T) didn't notify the server (UNRESOLVED_ROUTINE
> in SQL). Now sends a RegisterFunction command with the cloudpickled PythonUDF.
> Verified live: pandas_udf scalar/iterator/grouped-agg,
> applyInPandas/mapInPandas/mapInArrow/applyInArrow/cogroup, register+SQL all
> return correct results.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
