andygrove opened a new issue, #23894:
URL: https://github.com/apache/datafusion/issues/23894
### Describe the bug
`datafusion-spark`'s `mod` shares the `try_rem` helper in
`datafusion/spark/src/function/math/modulus.rs` with `pmod`. That helper has
two
gaps in its zero divisor handling. Both are visible through `mod`.
1. **ANSI mode, floating point divisor.** `try_rem` delegates to Arrow's
`rem`
kernel when `datafusion.execution.enable_ansi_mode` is `true`. Arrow's
`rem`
only reports division by zero for integer and decimal types. Floating
point
divisors follow IEEE 754 and quietly produce `NaN`. Spark raises
`REMAINDER_BY_ZERO` for a zero divisor of any numeric type.
2. **`-0.0` divisor, both modes.** The legacy path nulls out zero divisors by
comparing with `eq(right, 0)`. Arrow's comparison kernels order floating
point values totally, so `-0.0` is not equal to `0.0` and a `-0.0`
divisor is
not recognised. Spark's `isZero` check is a numeric comparison, so `-0.0`
counts as zero.
`pmod` was fixed for both cases as part of an audit of that function and no
longer routes through `try_rem`. `mod` still does.
### To Reproduce
Spark 4.2.0 (verified with `pyspark==4.2.0`):
```sql
SET spark.sql.ansi.enabled = true;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012
SET spark.sql.ansi.enabled = false;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE)); -- NULL
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE)); -- NULL
```
DataFusion:
```sql
set datafusion.execution.enable_ansi_mode = true;
SELECT mod(10.5::float8, 0.0::float8); -- NaN, expected an error
SELECT mod(10.5::float8, -0.0::float8); -- NaN, expected an error
set datafusion.execution.enable_ansi_mode = false;
SELECT mod(10.5::float8, -0.0::float8); -- NaN, expected NULL
```
### Expected behavior
`mod` raises for a zero divisor of any numeric type in ANSI mode and returns
`NULL` in legacy mode, treating `-0.0` as zero in both.
### Affected versions
Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 all behave the same way here. The
behaviour
difference across those versions is only the error class name, which changed
from `DIVIDE_BY_ZERO` to `REMAINDER_BY_ZERO` in Spark 4.1.
### Relevant code
`datafusion/spark/src/function/math/modulus.rs`, `try_rem`.
`spark_pmod` in the same file shows one possible shape for the fix. Its
`is_zero` helper treats `-0.0` as zero and its ANSI branch masks the check by
the validity of the dividend, because Spark's remainder expressions are null
intolerant and a NULL dividend short circuits to NULL before the divisor is
validated.
Surfaced by the audit-datafusion-spark-expression skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
