andygrove opened a new issue, #7211:
URL: https://github.com/apache/datafusion-comet/issues/7211
### Describe the bug
`CAST(ntz AS TIMESTAMP)` fails the task when a `TIMESTAMP_NTZ` value is at
or near the ends of the microsecond range:
```
org.apache.comet.CometNativeException: Cast error: Cannot convert
TimestampMicrosecondType 9223372036854775807 to datetime. Comet only supports
dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
```
Spark returns a value. In a UTC session, `Long.MAX` micros (`+294247-01-10
04:00:54.775807`) casts to the same instant, and `Long.MIN` micros
(`-290308-12-21 19:59:05.224192`) does too. By the message, any value outside
years -262144 to 262143 fails. The error comes from the range check in
`native/spark-expr/src/utils.rs`.
Spark inserts this cast implicitly, so other expressions over an NTZ column
fail the same way. For example, `date_trunc('MICROSECOND', ntz)` becomes
`date_trunc(MICROSECOND, cast(ntz as timestamp), Some(UTC))` and fails with the
same error. For coarser formats such as `HOUR`, Spark raises `long overflow` at
these values too, so only the error differs there.
The 1.1.0 release has the same bug: the cast was native and `Compatible`
there, and the check in `utils.rs` is unchanged.
### Steps to reproduce
```scala
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.sql("""SELECT CAST(timestamp_micros(id) AS TIMESTAMP_NTZ) AS ntz
FROM VALUES (9223372036854775807L), (-9223372036854775808L), (0L) AS
t(id)""")
.write.parquet("/tmp/ntz_ext")
spark.read.parquet("/tmp/ntz_ext").selectExpr("CAST(CAST(ntz AS TIMESTAMP)
AS STRING)").collect()
```
- Spark: `1970-01-01 00:00:00`, `-290308-12-21 19:59:05.224192`,
`+294247-01-10 04:00:54.775807`
- Comet: the task fails with the error above. Writing `CAST(ntz AS
TIMESTAMP)` to Parquet fails the same way.
`unix_micros(CAST(ntz AS TIMESTAMP))` matches Spark, because `unix_micros`
goes through the codegen dispatcher, which evaluates the cast in Spark's code.
### Expected behavior
The same values as Spark. Where Spark itself would overflow (a non-UTC
session near the range ends), the same error Spark raises.
### Additional context
Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression.
#5456 is the analogous gap for date-to-timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
