[
https://issues.apache.org/jira/browse/SPARK-59089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109699#comment-18109699
]
Pratham Manja commented on SPARK-59089:
---------------------------------------
[~holden] I wanted to work on this task. Could you please assign this Jira to
me? Thanks!
> Recover FloatType columns for the transpiler by widening operands to double
> ---------------------------------------------------------------------------
>
> Key: SPARK-59089
> URL: https://issues.apache.org/jira/browse/SPARK-59089
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Holden Karau
> Priority: Trivial
>
> SPARK-55210 made FloatType columns fall back to interpreted Python entirely,
> because an
> expression that stays in FloatType rounds to 24 bits per step where Python
> computes in double.
> That is more conservative than exactness requires: casting each float operand
> to DoubleType
> reproduces CPython exactly, so those columns could keep their lowering.
> ## Why
> FloatType is common in ML feature tables, and the current rule costs every
> such column its
> lowering permanently.
> Measured:
> - float32 -> Python float -> float32 round-trips **exactly** on 199163/199163
> finite bit
> patterns. So the value the interpreted UDF receives *is* the double a
> widened expression would
> compute with.
> - Staying in FloatType diverged from CPython on **20000/20000** random pairs
> for `(x + y) * y`.
> - Casting each operand to DoubleType and computing in double matched CPython
> on **20000/20000**
> — for a DoubleType declared return type *and* for a FloatType one, because
> `EvaluatePython.makeFromJava` narrows the interpreted double with a single
> `c.toFloat`.
> - Even the saturation case agrees: the double product 2.5373334837038975e44
> under one `.toFloat`
> is Infinity, the same as a trailing `cast(..., float)`.
> The objection recorded in the current comment — "would hide the rounding for
> a single operation
> but not for a chain of them, and not the overflow at all" — is about
> declaring a FloatType
> *return* type while still computing in FloatType. It does not apply to
> computing in double.
> ## How
> Admit FloatType to the "fractional" category in
> `ResolveTranspiledPythonUDFOptions`, and have the
> fractional variant wrap each parameter reference in `.cast("double")` — a
> no-op on a DoubleType
> column that `SimplifyCasts` removes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
