Haiyang Sun created SPARK-59745:
-----------------------------------
Summary: Support named arguments as in Python UDFs for unified
external UDF execution
Key: SPARK-59745
URL: https://issues.apache.org/jira/browse/SPARK-59745
Project: Spark
Issue Type: Sub-task
Components: UDF
Affects Versions: 4.4.0
Reporter: Haiyang Sun
The classic Python UDF path preserves keyword names through ArgumentMetadata,
but ExecuteExternalUDFExec currently rejects NamedArgumentExpression because
its temporary initialization contract only supports positional inputs.
Add named-argument support by:
- Forwarding each argument’s input offset and optional name to the worker.
- Preserving metadata through planning and expression rewrites.
- Binding reordered and keyword-only arguments correctly.
- Removing the current named-argument rejection.
- Adding tests for positional, reordered named, keyword-only, and mixed
arguments.
This should build on the language-agnostic initialization work tracked by
SPARK-59364. Named arguments must never silently fall back to positional
binding.
from pyspark.sql.functions import col, udf
@udf("int")
def subtract(a, b):
return a - b
df = spark.createDataFrame([(10, 3)], ["left", "right"])
df.select(
subtract(
b=col("right"),
a=col("left"),
).alias("result")
).show()
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
