andygrove opened a new issue, #6331:
URL: https://github.com/apache/datafusion-comet/issues/6331
### Describe the bug
Spark converts between instants and local time using the JVM's timezone
rules (`tzdb.dat`), whose version depends on the JDK build and on whether
TZUpdater has been run. Comet's native kernels use chrono-tz instead, which
compiles its own copy of the IANA database into libcomet. chrono-tz 0.10.4
carries 2025b. When the two versions disagree about a zone, every local-time
operation on the affected timestamps gives a different answer, and nothing
reports it. That covers `hour`, casts to string and date, `date_trunc`, parsing
strings into timestamps, and date-to-timestamp casts.
The drift can go either way. A JDK older than 2025b disagrees with Comet
about recent rule changes. A JDK newer than 2025b will disagree about whatever
changes after it, until we upgrade chrono-tz.
### Steps to reproduce
With AppleJDK 17.0.10, which ships tzdata 2023c, on `main` at `764936187`:
```sql
CREATE TABLE py USING parquet AS SELECT * FROM VALUES
(TIMESTAMP'2025-06-15T12:00:00Z') AS v(ts);
CREATE TABLE kz USING parquet AS SELECT * FROM VALUES
(TIMESTAMP'2024-06-30T23:30:00Z') AS v(ts);
SET spark.sql.session.timeZone=America/Asuncion;
SELECT hour(ts), CAST(ts AS STRING) FROM py;
SET spark.sql.session.timeZone=Asia/Almaty;
SELECT hour(ts), CAST(ts AS STRING) FROM kz;
```
For Asuncion, Spark returns `8, 2025-06-15 08:00:00` and Comet returns `9,
2025-06-15 09:00:00`. For Almaty, Spark returns `5, 2024-07-01 05:30:00` and
Comet returns `4, 2024-07-01 04:30:00`. Paraguay moved to permanent UTC-3
(tzdata 2025a), and Kazakhstan unified on UTC+5 on 2024-03-01 (tzdata 2024a).
2023c has neither change.
### Expected behavior
The same local times as Spark. Failing that, a documented limitation and a
warning when the two versions differ.
### Additional context
We could expose `chrono_tz::IANA_TZDB_VERSION` over JNI and log a warning at
startup when it differs from the JVM's version
(`ZoneRulesProvider.getVersions("UTC").lastKey()`). We could also document the
drift next to the existing chrono-tz horizon note in the datetime compatibility
guide, and keep chrono-tz current. #4754 is related, though jiff reading the
system tz database wouldn't match the JVM's bundled copy either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
