ueshin commented on code in PR #50759:
URL: https://github.com/apache/spark/pull/50759#discussion_r2096314380
##########
python/pyspark/sql/pandas/_typing/__init__.pyi:
##########
@@ -59,8 +59,81 @@ PandasGroupedMapUDFTransformWithStateType = Literal[211]
PandasGroupedMapUDFTransformWithStateInitStateType = Literal[212]
GroupedMapUDFTransformWithStateType = Literal[213]
GroupedMapUDFTransformWithStateInitStateType = Literal[214]
+ArrowScalarUDFType = Literal[215]
class PandasVariadicScalarToScalarFunction(Protocol):
+ def __call__(self, *_: pyarrow.Array) -> pyarrow.Array: ...
Review Comment:
This should be `ArrowVariadicScalarToScalarFunction`?
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -66,6 +66,7 @@ private[spark] object PythonEvalType {
val SQL_TRANSFORM_WITH_STATE_PANDAS_INIT_STATE_UDF = 212
val SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_UDF = 213
val SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_INIT_STATE_UDF = 214
+ val SQL_SCALAR_ARROW_UDF = 215
Review Comment:
nit: how about using a separate id group, like `250` and above, following
the corresponding pandas UDF's eval type, like `SQL_GROUPED_MAP_ARROW_UDF =
251`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
