L. C. Hsieh created SPARK-60119:
-----------------------------------
Summary: Casting a string to decimal fails or is slow when the
exponent is far outside the target scale
Key: SPARK-60119
URL: https://issues.apache.org/jira/browse/SPARK-60119
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
`Decimal.changePrecision` rescales its BigDecimal value with `setScale` without
bounding the scale change. Its cost is proportional to the change, and it
throws once the change exceeds the Int range:
1. `CAST('1e-2147483647' AS DECIMAL(10,2))` throws
`java.lang.ArithmeticException: BigInteger would overflow supported range`
under LEGACY, ANSI and TRY, instead of returning 0.00. `'1e-100000000'` returns
0.00 but takes ~40s, since it materializes 10^100000000.
2. With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, `fromString`
intentionally skips the integral-digit fast-fail, so `CAST('1e2147483647' AS
DECIMAL(10,2))` throws `java.lang.ArithmeticException: Underflow` instead of
returning NULL (non-ANSI/TRY) or `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI).
`'1e100000000'` is also very slow. When `changePrecision` does report the
overflow under ANSI, the error message renders the value with `toPlainString`,
which cannot fit `1E+2147483647` in a String.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
