David Mollitor created SPARK-59719:
--------------------------------------
Summary: Mark CheckOverflow as null-intolerant to match MakeDecimal
Key: SPARK-59719
URL: https://issues.apache.org/jira/browse/SPARK-59719
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. Summary
{{CheckOverflow}} does not declare {{nullIntolerant}}, unlike its sibling
{{MakeDecimal}}
(and {{UnscaledValue}}, {{Cast}}, and the arithmetic expressions). It already
satisfies the null-intolerance contract at runtime -- it extends
{{UnaryExpression}} and overrides
{{nullSafeEval}}, so {{UnaryExpression.eval}} returns null whenever the child
is null -- so the property is simply undeclared. When the {{NullIntolerant}}
marker trait was replaced by a {{def}} (SPARK-50241), the adjacent
{{UnscaledValue}} and {{MakeDecimal}} were updated but {{CheckOverflow}} was
missed.
h3. Change
Declare it null-intolerant:
{code:scala}
override def nullIntolerant: Boolean = true
{code}
h3. Why
{{nullIntolerant}} is consumed by the optimizer to:
* fold a null-literal input to a null literal ({{NullPropagation}}), and
* infer {{IsNotNull(child)}} constraints and push down {{IsNotNull}} filters
({{QueryPlanConstraints.scanNullIntolerantAttribute}} /
{{InferFiltersFromConstraints}}).
{{CheckOverflow}} wraps decimal results throughout the plan (e.g. the decimal
serializers in {{SerializerBuildHelper}}), so declaring the property it already
honors lets these optimizations see through it, consistent with {{MakeDecimal}}.
The direction the optimizer relies on holds even though {{CheckOverflow}} with
{{nullOnOverflow = true}} can also return null from a non-null input on
overflow: the contract only requires "null in => null out", not the converse --
the same situation as {{Cast}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
