andygrove opened a new issue, #6426: URL: https://github.com/apache/datafusion-comet/issues/6426
### Describe the bug #5638 added native kernels for Iceberg's `years`, `months`, `days` and `hours` functions, and they run by default. #5773 also routes these functions to the native kernels when Iceberg's SQL extensions rewrite them to `ApplyFunctionExpression`. The kernels compute a true floor. Iceberg's Java `DateTimeUtil` returns one unit less for some timestamps: a negative timestamp whose microseconds are 999999 at a unit boundary. Iceberg's Spark functions call `DateTimeUtil`, so Spark returns Iceberg's value, and it is also the value Iceberg partitions by. In 1.0.0 these functions ran in Spark, so this is a regression in 1.1.0. It reproduces on 1.1.0-rc1, and `main` has the same kernels. ### Steps to reproduce With an Iceberg catalog `ice`, the session timezone set to UTC, and a table `t` holding `TIMESTAMP '1969-01-01 00:00:00.999999'`, run: ```sql SELECT ice.system.years(ts), ice.system.months(ts), ice.system.days(ts), ice.system.hours(ts) FROM t ``` With Iceberg's SQL extensions installed, a filter such as `WHERE hours(ts) = -2` goes through the same kernels. ### Expected behavior Spark with Iceberg, and Comet 1.0.0, return `-2, -13, 1968-12-31, -8761`. Comet 1.1.0-rc1 returns `-1, -12, 1969-01-01, -8760`. For a table with `1969-12-31 23:00:00.999999` and `1969-12-31 22:30:00`, `WHERE hours(ts) = -2` returns both rows in Spark and only the second on rc1. `TIMESTAMP_NTZ` inputs behave the same way. ### Additional context For negative input, `DateTimeUtil.convertMicros` builds the instant from two parts: `floorDiv(micros, 1_000_000)` seconds and `floorMod(micros + 1, 1_000_000)` microseconds. It then subtracts one from the unit count. At these boundaries that undercounts by one. Iceberg 1.5.2 through 1.11.0 all behave this way. The native kernels in `native/spark-expr/src/iceberg_funcs/temporal.rs` need to mirror it to agree with Iceberg. The inputs that trigger it are rare, but the result is wrong silently. Found by the 1.1.0 regression audit (#6399) and tracked in #6402. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
