viirya opened a new issue, #6966:
URL: https://github.com/apache/datafusion-comet/issues/6966
### Describe the bug
Spark parses a string to decimal with `new java.math.BigDecimal(str)` and
then rescales it to the target type with `BigDecimal.setScale`. For extreme
exponents that rescale fails with a raw `java.lang.ArithmeticException` in
every eval mode (ANSI, legacy and `try_cast`), while the native cast returns a
value or NULL. Found while reviewing #6826.
Two shapes:
1. **Positive target scale.** `'1e-2147483647'` cast to `DECIMAL(10,2)`: the
parsed scale (2147483647) fits in an int, so the string is valid, but the scale
difference to the target is about 2.1e9. Spark fails with `BigInteger would
overflow supported range`. The native cast returns `0.00`, because
`parse_string_to_decimal` returns zero whenever the value has to be divided by
more than 10^38.
2. **Negative target scale**
(`spark.sql.legacy.allowNegativeScaleOfDecimal=true`). `'1e-2147483647'` cast
to `DECIMAL(10,-2)`: the scale difference does not fit in an int, and Spark
fails with `Underflow`. Since #6826 the native cast reports
`CAST_INVALID_INPUT` under ANSI and NULL otherwise, so both engines fail under
ANSI but with different exceptions, and they differ outside ANSI.
On JDK 17 and 21, `setScale` returns zero for scale differences up to about
1e8 (after computing a very large power of ten) and throws from about 6e8. The
exact threshold depends on the JDK's `BigInteger` limits, so the native cast
cannot reproduce it exactly.
### Steps to reproduce
```sql
CREATE TABLE t(s string) USING parquet;
INSERT INTO t VALUES ('1e-2147483647');
SELECT try_cast(s AS DECIMAL(10,2)) FROM t;
```
Spark: `java.lang.ArithmeticException: BigInteger would overflow supported
range`. Comet: `0.00`.
### Expected behavior
To be decided. Spark raising a raw `ArithmeticException` even from
`try_cast` looks like a Spark-side issue. Comet should at least not silently
return a value where Spark fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
