manesioz opened a new pull request, #377: URL: https://github.com/apache/doris-spark-connector/pull/377
## Problem Closes #376. Doris `DATETIME` and `DATETIMEV2` store wall-clock values without timezone semantics. At times, Doris BE nodes can return them through timezone-aware Arrow vectors. `RowBatch` treated these values like `TIMESTAMPTZ` instants. When the Arrow timezone differed from the Spark JVM timezone, the displayed clock shifted. For example, Doris `2026-08-24 10:00:09` could become `2026-08-24 02:00:09` when Spark ran in UTC but the BE configured tz was Shanghai/UTC+8. [#366](https://github.com/apache/doris-spark-connector/pull/366) supports Arrow timestamps without timezone metadata. It does not recover wall-clock values when older backends include this metadata. ## Solution - Separate `DATETIME` and `DATETIMEV2` conversion from `TIMESTAMPTZ` conversion. - Decode timezone-aware `DATETIME` values with the Arrow timezone and declared timestamp unit. - Preserve the existing `LocalDateTime` path when the Arrow timezone is null. - Keep `TIMESTAMPTZ` on its existing instant-preserving path. - Resolve timezone metadata once per column instead of once per row. - Update integration expectations so Thrift and Arrow reads return the stored wall-clock value. This supports older backends that include timezone metadata and newer backends that return timezone-naive timestamps. ## Tests Regression coverage verifies: - `DATETIME` and `DATETIMEV2` with `+08:00` Arrow metadata. - Null Arrow timezone metadata. - Second, millisecond, microsecond, and nanosecond units. - Negative epoch values. - Different semantics for identical `DATETIME` and `TIMESTAMPTZ` Arrow values. - Both Java `Timestamp` and Java 8 `Instant` output modes. Validation completed: - [x] Spark 3.5 `RowBatchTest` in UTC: 16 tests passed. - [x] Spark 3.5 `RowBatchTest` in Asia/Shanghai: 16 tests passed. - [x] Spark 4.1 `RowBatchTest` in UTC: 16 tests passed. - [x] Spark 3.5 integration test sources compiled. - [x] `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
