andygrove opened a new issue, #3202: URL: https://github.com/apache/datafusion-comet/issues/3202
## Description The `date_format` expression was added in PR #3201, but currently only supports UTC timezone with full compatibility. Non-UTC timezones are marked as `Incompatible` and fall back to Spark by default. This issue tracks adding proper timezone conversion support so that `date_format` can be fully compatible with Spark for all timezones. ## Current Behavior - UTC timezone: `Compatible()` - runs natively in Comet - Non-UTC timezones: `Incompatible()` - falls back to Spark by default - Users can enable non-UTC with `spark.comet.expr.DateFormatClass.allowIncompatible=true` but results may differ from Spark ## Desired Behavior All timezones should be `Compatible()` and produce results identical to Spark. ## Technical Details The current implementation uses DataFusion's `to_char` function which formats timestamps without timezone conversion. Spark's `date_format` applies the session timezone when formatting. Possible approaches: 1. Convert the timestamp to the target timezone before calling `to_char` 2. Use a timezone-aware formatting function if available in DataFusion 3. Implement custom Rust logic to handle timezone conversion ## Related - PR #3201 - Initial date_format implementation - Issue #2649 - Similar timezone issue with `date_trunc` --- > **Note:** This issue was generated with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
