andygrove opened a new pull request, #4821: URL: https://github.com/apache/datafusion-comet/pull/4821
## Summary - Wires the Spark 4.1 `TIME` data type end-to-end and adds JVM codegen dispatch for the new TIME expressions (`HoursOfTime`, `MinutesOfTime`, `SecondsOfTime`, `SecondsOfTimeWithFraction`, `TimeAddInterval`, `SubtractTimes`, `TimeDiff`, `TimeTrunc`) and casts touching `TimeType`. No native implementations added. - Teaches the batch-kernel input/output emitters, `CometScalaUDFCodegen`, and `ArrowWriters` to handle `TimeNanoVector` as a Long-backed vector. - Parquet scans still opt out of `TimeType` (no native decoder yet); the fallback reason is now explicit. ## Details Every new TIME expression in Spark 4.1 is a `RuntimeReplaceable` that rewrites to `StaticInvoke(DateTimeUtils, ...)` before physical planning. The shim (`spark-4.1+/CometExprShim.scala`) matches those `StaticInvoke` forms and routes them through `CometScalaUDF.emitJvmCodegenDispatch`, which runs Spark's own `doGenCode` inside the Comet batch kernel. `CurrentTime` needs no work — the optimizer folds it to a literal. `CometCast` routes any cast touching `TimeType` through the same dispatcher. Existing `make_time` / `to_time` / `try_to_time` continue to use the native scalar-function paths already wired up. `LocalTableScan` for `TimeType` columns now works natively via the new `TimeNanoWriter` in `ArrowWriters`. The pre-existing "falls back when schema contains TimeType" test flips to "handles TimeType column". Known limitation: `time - time` (result type `DayTimeIntervalType`) still falls back at the projection level because Comet's generic type-support list doesn't cover `DayTimeIntervalType` outputs. Pre-existing, not TIME-specific — left as a `checkSparkAnswer` regression test in the suite. ## Test plan - [x] `./mvnw test -Dsuites="org.apache.spark.sql.CometTimeTypeSuite" -Dtest=none` — 15/15 pass on Spark 4.1 (new suite) - [x] `./mvnw test -Dsuites="org.apache.comet.exec.CometExecSuite" -Dtest=none` — 131/131 pass (LocalTableScan regression check) - [x] `./mvnw test -Dsuites="org.apache.comet.CometCastSuite" -Dtest=none` — 161/161 pass (cast regression check) - [x] `./mvnw test -Dsuites="org.apache.comet.CometCodegenSourceSuite" -Dtest=none` — 54/54 pass (codegen kernel regression check) - [x] `./mvnw test -Dsuites="org.apache.comet.CometExpressionSuite" -Dtest=none` — 135/135 pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
