[
https://issues.apache.org/jira/browse/SPARK-61163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61163.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 352
[https://github.com/apache/spark-connect-rust/pull/352]
> Make GroupedData.apply take only pandas UDFs of SQL_GROUPED_MAP_PANDAS_UDF,
> and warn, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61163
> URL: https://issues.apache.org/jira/browse/SPARK-61163
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's GroupedData.apply takes only a pandas_udf of
> SQL_GROUPED_MAP_PANDAS_UDF: anything else, such as a function, a UDF of
> another eval type or a Column (which has any attribute), raises
> PySparkTypeError INVALID_UDF_EVAL_TYPE of eval_type
> SQL_GROUPED_MAP_PANDAS_UDF. It applies such a UDF with applyInPandas, with a
> UserWarning to prefer applyInPandas.
> The Python client checks only that the argument has a func attribute: it
> raises a plain TypeError ("apply() expects a pandas_udf of type
> SQL_GROUPED_MAP_PANDAS_UDF") for a function, sends a UDF of another eval type
> or a Column's field to applyInPandas, and does not warn. Apache Spark's
> test_wrong_args of test_parity_pandas_grouped_map.py fails on this client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
