dwsmith1983 opened a new issue, #6558: URL: https://github.com/apache/datafusion-comet/issues/6558
### Describe the bug #5365 adds calendar rebasing to the native Parquet read path for the Delta contrib scan (the regular native scan still has none, #5010). When that path refuses ancient dates or timestamps under `spark.sql.parquet.datetimeRebaseModeInRead=EXCEPTION` (or the INT96 equivalent), it raises `org.apache.comet.CometNativeException` with "Native scan cannot rebase ancient values in column ...". Spark raises `org.apache.spark.SparkUpgradeException` (`INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME`) for the same file, so code that catches Spark's upgrade error, or checks its error class, sees a different exception. The values are not affected: the native scan refuses where Spark refuses. Only the exception type differs. ### Steps to reproduce Read a Parquet file written by a legacy writer, or one with no Spark writer metadata, that holds a date before 1582-10-15, through the native Delta scan with `spark.sql.parquet.datetimeRebaseModeInRead=EXCEPTION`. Spark with Comet off raises `SparkUpgradeException`; the native scan raises `CometNativeException`. ### Expected behavior The native refusal surfaces as the `SparkUpgradeException` Spark raises, with the same error class and message parameters, on every supported Spark version. ### Additional context Comet has no native error that maps to `SparkUpgradeException` yet. The constructor differs across Spark 3.4 to 4.2, so the conversion goes through `ShimSparkErrorConverter`. The same mapping will serve the regular native scan once it rebases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
