[
https://issues.apache.org/jira/browse/SPARK-59069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59069:
-----------------------------------
Labels: pull-request-available (was: )
> Transpile Python UDFs with explicit positional-only parameters
> --------------------------------------------------------------
>
> Key: SPARK-59069
> URL: https://issues.apache.org/jira/browse/SPARK-59069
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Holden Karau
> Priority: Trivial
> Labels: pull-request-available
>
> The transpiler refuses a UDF with positional-only parameters outright:
> ```python
> def f(a, b, /):
> return a + b
> ```
> falls back to interpreted Python, alongside defaults, `*args`, `**kwargs` and
> keyword-only parameters. For those four the refusal is necessary: the
> transpiled
> expression references its inputs positionally through `_udf_param_N`, and a
> call site
> that omits a defaulted argument, or passes one by keyword only, leaves a
> placeholder
> pointing at a position nothing bound.
> A positional-only parameter has no such gap. It is *always* bound by position
> -- that is
> what positional-only means -- so the placeholder scheme fits it exactly. The
> refusal was
> over-broad, and it came about only because `_get_parameter_list` read
> `node.args.args` and never `node.args.posonlyargs`, so the parameter list
> would simply
> have been short by however many positional-only parameters the function
> declared. Refusing
> was the safe response to that, not a statement about the feature.
> Two parts:
> **1. Lower them.** Read `posonlyargs + args` everywhere the positional
> parameter list is
> derived -- `_get_parameter_list`, `_param_category_combos` (so each
> parameter's
> input-type category lands in the right slot), and the located-versus-held
> lambda
> parameter comparison in `_get_function_from_ast`. A shared `_positional_args`
> helper
> keeps the concatenation in one place. Drop `posonlyargs` from the refusal
> list; a
> *defaulted* positional-only parameter still hits the `defaults` check, as it
> must.
> **2. Do not resolve a positional-only kwarg to a slot.**
> `UserDefinedFunction.__call__`
> rewrites user kwargs to positional order, so that a transpiled `_udf_param_N`
> expression
> sees plain column references rather than a `NamedArgumentExpression` (which
> breaks nested
> calls such as `isnotnull`). Once part 1 makes these UDFs candidates, that
> rewrite would
> happily resolve `f(a=col)` for a positional-only `a` -- silently turning a
> call **Python
> itself rejects** into a valid one:
> ```python
> def f(a, /):
> return a + 1
> f(a=1) # TypeError: f() got some positional-only arguments passed
> as keyword arguments
> udf_f(a=col) # would have succeeded, with transpilation on
> ```
> So track which public parameters are positional-only and exclude them from
> the rewrite.
> Left unresolved, the kwarg reaches the JVM as a `NamedArgumentExpression`,
> which drops the
> transpiled expression and routes to the interpreted path -- where the
> worker's own keyword
> call raises the same `TypeError` Python would. The fallback does the right
> thing; it just
> has to be allowed to happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
