andygrove opened a new issue, #23896:
URL: https://github.com/apache/datafusion/issues/23896
### Describe the bug
Spark's `Pmod` declares `inputType = NumericType` and relies on the
analyzer's
string promotion rule, so a string argument is implicitly cast. DataFusion's
`SparkPmod` uses `Signature::numeric(2, Volatility::Immutable)`, which
rejects
strings at planning time.
### To Reproduce
Spark 4.2.0 (verified with `pyspark==4.2.0`), and covered by Spark's own
golden
file
`sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out`:
```sql
SELECT pmod('1', CAST(1 AS TINYINT)); -- 0, result type bigint
SELECT pmod('10', CAST(3 AS INT)); -- ansi on: 1 (int); ansi off: 1.0
(double)
SELECT pmod('1', '1'); --
DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE
```
Note that Spark's promotion target differs by mode: with
`spark.sql.ansi.enabled = true` the string is cast to the other argument's
type,
and with it disabled the pair is promoted to `double`. Two strings are
rejected
in both modes.
DataFusion:
```sql
SELECT pmod('10'::string, 3::int);
-- Error during planning: For function 'pmod' Utf8 and Int32 are not
coercible
-- to a common numeric type.
```
### Expected behavior
A string argument is accepted and coerced the way Spark coerces it.
### Scope
This is not specific to `pmod`. Every `datafusion-spark` function that uses
`Signature::numeric` inherits it, and reproducing Spark's mode-dependent
promotion target needs a decision about how far `datafusion-spark` should go
in
emulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a `pmod`-local workaround.
A commented-out query recording the Spark 4.2.0 result is in
`datafusion/sqllogictest/test_files/spark/math/pmod.slt`.
### Relevant code
`datafusion/spark/src/function/math/modulus.rs`, `SparkPmod::new`.
Surfaced by the audit-datafusion-spark-expression skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
