[
https://issues.apache.org/jira/browse/SPARK-59880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Holden Karau resolved SPARK-59880.
----------------------------------
Resolution: Duplicate
Duplicate of SPARK-55219
> Broaden transpilation input type categories to timestamp, map, and array
> ------------------------------------------------------------------------
>
> Key: SPARK-59880
> URL: https://issues.apache.org/jira/browse/SPARK-59880
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 4.4.0
> Reporter: Holden Karau
> Priority: Major
>
> `_annotation_category` knows `str`, `int`, `float`, `bool`, `bytes`. Anything
> else comes back `None`, which means untyped, so `_param_category_combos` tries
> the parameter as numeric *and* string and `_transpile_func` tags each option
> with
> that guess. `optionMatchesTypes` is strict and ends in `case _ => false`, so a
> `datetime` parameter gets matched `("numeric", TimestampType)` and
> `("string", TimestampType)`, both fail, every option is pruned, and the Python
> UDF stays.
>
> **Fix.** Teach the categories the types a variety can lower, and leave the
> matcher strict:
> - `python/pyspark/sql/transpile.py:937` `_annotation_category` — `datetime` →
> `"timestamp"`, `dict[str, str]` → `"map"`, `list[X]` → `"array"`. Pinning
> them
> also shrinks the combo matrix, since they stop counting as untyped.
> - `python/pyspark/sql/transpile.py:960` `_param_category_combos` — nothing to
> do
> once the annotations resolve; the `untyped > 3` collapse just fires less.
> - `.../catalyst/analysis/ResolveTranspiledPythonUDFOptions.scala:95`
> `optionMatchesTypes` — add the arms, keep `case _ => false`.
> Granularity is the actual decision. `"array"` matching any `ArrayType` lets an
> `array<string>` lowering match an `array<int>` column; `"array<string>"` is
> precise and starts putting a type grammar inside a category string. J2 makes
> the
> question go away, and then this ticket is three lines in
> `_annotation_category`.
> We should add one UDF per new category in `test_udf_transpile_unit.py`,
> asserting the
> option survives against a matching column type and is pruned against a
> non-matching one. The built-in transpiler still refuses these bodies, so
> assert
> on `_transpiled_input_categories` or stand up a stub variety.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
