snmvaughan opened a new issue, #7187: URL: https://github.com/apache/datafusion-comet/issues/7187
### Describe the bug Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), include [SPARK-60119](https://issues.apache.org/jira/browse/SPARK-60119) (apache/spark#59335). When the scale change is over 38, `Decimal.changePrecision` no longer calls `BigDecimal.setScale` if the value's magnitude already decides the result: a value below 0.01 ulp of the target scale is rounded as 0.01 ulp of the same sign. With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, that changes the negative-scale case of #6966. `'1e-2147483647'` cast to `DECIMAL(10,-2)` returns 0 in every eval mode, where earlier releases throw `ArithmeticException: Underflow`. The native cast still returns NULL, or raises `CAST_INVALID_INPUT` under ANSI, because `parse_string_to_decimal` rejects a scale difference that doesn't fit in an int. The positive-scale case of #6966 now matches: for `'1e-2147483647'` cast to `DECIMAL(10,2)`, Spark returns `0.00`, as the native cast already does (see #7171). ### Steps to reproduce On a Spark build with SPARK-60119 (branch-4.0, branch-4.1, branch-4.2 or the release candidates above), with Comet enabled: ```sql SET spark.sql.legacy.allowNegativeScaleOfDecimal=true; CREATE TABLE t (s STRING) USING parquet; INSERT INTO t VALUES ('1e-2147483647'); SELECT try_cast(s AS DECIMAL(10, -2)) FROM t; -- Spark: 0 -- Comet: NULL ``` With ANSI enabled, `CAST(s AS DECIMAL(10, -2))` returns 0 in Spark and raises `CAST_INVALID_INPUT` in Comet. ### Expected behavior On releases with SPARK-60119, Comet returns what Spark does: the rounded value, 0 here, for a string far below the target scale's ulp. Earlier releases throw a raw `ArithmeticException`, which the native cast can't reproduce, so it would keep failing under ANSI and returning NULL otherwise there. ### Additional context SPARK-60119 leaves nothing in the plan for Comet to detect, so the fix would choose the behavior by Spark version, as #7171 does for SPARK-58821 and SPARK-60118 in `SparkFixes`. All three are in the same releases. `CometNativeCastSuite`'s `(ansi) cast string to negative-scale decimal fails when rescaling overflows an int` expects Spark to fail on this input, so it fails against builds with SPARK-60119. ### Willingness to contribute I can contribute a fix for this bug independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
