andygrove opened a new issue, #7463: URL: https://github.com/apache/datafusion-comet/issues/7463
### What is the problem the feature request solves? Found in review of #7053. The native Parquet writer fallback added there, `NativeWriteUtils.legacyDatetimeRebaseWriteReason`, treats `spark.sql.parquet.int96RebaseModeInWrite=LEGACY` as a reason to fall back whenever the output has a timestamp column. That setting only governs the INT96 encoding. When `spark.sql.parquet.outputTimestampType` is `TIMESTAMP_MICROS` or `TIMESTAMP_MILLIS`, Spark writes INT64 and uses `datetimeRebaseModeInWrite` instead, so the INT96 setting has no effect on the output. Reproduction: a timestamp column, `outputTimestampType=TIMESTAMP_MICROS`, `datetimeRebaseModeInWrite=CORRECTED`. Changing only `int96RebaseModeInWrite` from `CORRECTED` to `LEGACY` makes the helper return a fallback reason instead of `None`. Spark writes the same INT64 TIMESTAMP(MICROS) either way (for `1500-01-01 12:34:56.123456` both files contain `-14831724303876544`). The fallback is safe, but an unrelated legacy INT96 setting disables native write acceleration for a write the native writer handles correctly. ### Describe the potential solution Only apply the INT96 condition when `spark.sql.parquet.outputTimestampType` is `INT96`. The native writer always writes INT64 microseconds and ignores `outputTimestampType` when writing, so the check has to read the session conf. Add a regression test with a non-INT96 timestamp output and `int96RebaseModeInWrite=LEGACY` that asserts the write stays native and matches Spark's value. Update the paragraph in `docs/source/user-guide/latest/parquet.md` that names `int96RebaseModeInWrite=LEGACY` as a fallback trigger. ### Additional context Reviewer comment: https://github.com/apache/datafusion-comet/pull/7053#discussion_r4458640616 Follow-on to #7053, which should merge first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
