andygrove opened a new issue, #7216:
URL: https://github.com/apache/datafusion-comet/issues/7216
### Describe the bug
Under ANSI mode, when the native string to decimal cast rejects a value as
out of range with `NUMERIC_OUT_OF_SUPPORTED_RANGE`, such as `'0e38'`, Comet
throws a plain `org.apache.spark.SparkException`. Spark throws
`org.apache.spark.SparkArithmeticException`, from `outOfDecimalTypeRangeError`
(`DataTypeErrors` on 4.x, `QueryExecutionErrors` on 3.4 and 3.5). The error
class, SQLSTATE and message are the same. Only the exception type differs, so
code that catches `ArithmeticException` behaves differently with Comet enabled.
The native side raises `SparkError::NumericOutOfRange`, which
`exception_class()` in `native/common/src/error.rs` groups with the
`SparkArithmeticException` errors. But none of the three
`ShimSparkErrorConverter`s (`spark-3.4`, `spark-3.5`, `spark-4.x`) has a
`"NumericOutOfRange"` case. So `SparkErrorConverter.convertToSparkException`
falls through to its generic `new SparkException(errorClass = ..., ...)`. In
the 4.x shim, the only other native error type without a case is `Arrow`.
### Steps to reproduce
```sql
SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES ('0e38');
SELECT CAST(s AS DECIMAL(10, 2)) FROM t;
```
On `main` at 330b29cf10 with Spark 4.1.3, both engines fail with
`[NUMERIC_OUT_OF_SUPPORTED_RANGE] The value 0e38 cannot be interpreted as a
numeric since it has more than 38 digits. SQLSTATE: 22003`. Spark's exception
is a `SparkArithmeticException`, and Comet's is a `SparkException`.
A SQL file test with `query
expect_error_class(NUMERIC_OUT_OF_SUPPORTED_RANGE)` on that query fails with
`class org.apache.spark.SparkException did not equal class
org.apache.spark.SparkArithmeticException`.
### Expected behavior
Comet throws the same `SparkArithmeticException` as Spark. Adding a
`"NumericOutOfRange"` case to each shim that calls that version's
`outOfDecimalTypeRangeError` would do it, with the `expect_error_class` query
above as the test.
### Additional context
- `parse_string_to_decimal` in
`native/spark-expr/src/conversion_funcs/string.rs` is the only place that
raises `NumericOutOfRange` today, for a zero with too many integral digits.
#7188 would raise it for nonzero values with more than 38 integral digits too,
so they would hit the same fallback.
- The cast tests compare messages, not exception classes. `castTest` in
`CometNativeCastSuite` compares the start of the message on Spark 4, which is
why this hasn't shown up.
- Found while reviewing #7171.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
