Satyr09 opened a new pull request, #5789: URL: https://github.com/apache/datafusion-comet/pull/5789
## Which issue does this PR close? Closes #5577. ## Rationale for this change `unix_timestamp(string[, format])` currently falls back to a Spark projection even though Comet's codegen dispatcher can execute Spark's parser. Routing these inputs through the dispatcher keeps the projection in Comet and preserves Spark's parsing behavior. ## What changes are included in this PR? - Add `CodegenDispatchFallback` to `CometUnixTimestamp`. Keep the native input-type guard so `allowIncompatible=true` cannot send strings to the native date/timestamp kernel. - Replace the existing string fallback assertions with checks that require Comet execution and dispatcher activity. Cover collated strings and formats, including the native opt-in for date/timestamp inputs. - Strengthen parser-policy SQL tests and add coverage for column formats, nulls, invalid dates, ANSI errors, time zones, DST boundaries, dictionary encoding and disabled dispatch. - Update the expression support documentation and add a focused benchmark. The native implementation is reused without Rust changes. The duplicate input check mentioned in the issue was already removed by #5595. ## How are these changes tested? The full `CometTemporalExpressionSuite` and SQL files matching `unix_timestamp` pass on Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and experimental 4.2.0. Spark 3.4 and 3.5 each pass 55 tests and skip two Spark 4 collation tests. Each Spark 4 profile passes 57 tests. Removing only the new mixin makes all three dispatch regression tests fail while the three existing date/timestamp tests still pass. [Test and regression results](https://github.com/Satyr09/datafusion-comet/actions/runs/34320611637). Scalafix, Spotless, style and license checks, suite registration checks, benchmark runner checks and documentation generation pass on the Scala 2.12 and 2.13 profiles. Both formatting patches are empty. [Formatting and documentation results](https://github.com/Satyr09/datafusion-comet/actions/runs/34320611615). The native release library was built from the same native source tree as the tested commit. [Native build](https://github.com/Satyr09/datafusion-comet/actions/runs/34292740973). Release benchmark on Spark 4.0.4, JDK 17 and an AMD EPYC 7763 GitHub runner, using 1,048,576 rows: | Input | Spark best time | Comet best time | | --- | ---: | ---: | | String with default format | 739 ms | 672 ms | | String with a format column | 1,841 ms | 1,782 ms | | Native timestamp control | 91 ms | 52 ms | The benchmark checks that the expressions survive optimization and the Comet plans contain Comet operators. These timings come from one runner; the native timestamp case measures the existing implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
