[
https://issues.apache.org/jira/browse/SPARK-59719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59719:
----------------------------------
Affects Version/s: 4.4.0
(was: 4.1.0)
> Mark CheckOverflow as null-intolerant to match MakeDecimal
> ----------------------------------------------------------
>
> Key: SPARK-59719
> URL: https://issues.apache.org/jira/browse/SPARK-59719
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h3. Summary
> {{CheckOverflow}} does not declare {{nullIntolerant}}, unlike its sibling
> {{MakeDecimal}}
> (and {{UnscaledValue}}, {{Cast}}, and the arithmetic expressions). It already
> satisfies the null-intolerance contract at runtime -- it extends
> {{UnaryExpression}} and overrides
> {{nullSafeEval}}, so {{UnaryExpression.eval}} returns null whenever the child
> is null -- so the property is simply undeclared. When the {{NullIntolerant}}
> marker trait was replaced by a {{def}} (SPARK-50241), the adjacent
> {{UnscaledValue}} and {{MakeDecimal}} were updated but {{CheckOverflow}} was
> missed.
> h3. Change
> Declare it null-intolerant:
> {code:scala}
> override def nullIntolerant: Boolean = true
> {code}
> h3. Why
> {{nullIntolerant}} is consumed by the optimizer to:
> * fold a null-literal input to a null literal ({{NullPropagation}}), and
> * infer {{IsNotNull(child)}} constraints and push down {{IsNotNull}} filters
> ({{QueryPlanConstraints.scanNullIntolerantAttribute}} /
> {{InferFiltersFromConstraints}}).
> {{CheckOverflow}} wraps decimal results throughout the plan (e.g. the decimal
> serializers in {{SerializerBuildHelper}}), so declaring the property it
> already honors lets these optimizations see through it, consistent with
> {{MakeDecimal}}.
> The direction the optimizer relies on holds even though {{CheckOverflow}} with
> {{nullOnOverflow = true}} can also return null from a non-null input on
> overflow: the contract only requires "null in => null out", not the converse
> -- the same situation as {{Cast}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
