dwsmith1983 opened a new pull request, #6559:
URL: https://github.com/apache/datafusion-comet/pull/6559
## Which issue does this PR close?
Part of #6558.
## Rationale for this change
Comet has no native error that maps to `SparkUpgradeException`, so a native
rebase refusal under `datetimeRebaseModeInRead=EXCEPTION` (or the INT96
equivalent) surfaces as `CometNativeException`. This adds the mapping so the
refusal in #5365, and later the regular native scan (#5010), can raise the same
exception Spark does.
Nothing on main produces the new error yet. #5365 will switch its refusal to
it once this lands, and that change closes #6558.
## What changes are included in this PR?
- `SparkError::ReadAncientDatetime { format, column }` with error class
`INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME`. The
`read_ancient_datetime(column, is_int96)` constructor limits `format` to the
two values Spark uses for Parquet, "Parquet" and "Parquet INT96".
- A case in the 3.4, 3.5 and 4.x `ShimSparkErrorConverter` that calls
`DataSourceUtils.newRebaseExceptionInRead(format)`. That helper has the same
signature on 3.4 through 4.2 and picks the config and datasource option for the
format, so the message parameters match Spark's reader. The exception is
returned unwrapped, as Spark's `FileScanRDD` (3.x) and
`FileDataSourceV2.attachFilePath` (4.x) rethrow it without `FAILED_READ_FILE`.
A producer has to return the error as `DataFusionError::External` (or
`ParquetError::External`) for the JNI layer to send it as a
`CometQueryExecutionException`.
## How are these changes tested?
- Rust: the mapping tests in `error.rs` cover the error class, exception
class, type name and JSON params for both formats, and a `jni-bridge` test
checks that `DataFusionError::External(ReadAncientDatetime)` is classified as a
`CometQueryExecutionException` with that payload.
- `SparkErrorConverterSuite`: for both formats, conversion through
`convertErrorType` and from the native JSON through `convertToSparkException`,
asserting the class, error class and the `format`, `config` and `option`
parameters. An unknown format fails in Spark's helper instead of converting.
- Ran the suite on Spark 3.4, 3.5 (strict warnings), 4.0, 4.1 and 4.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
