manesioz opened a new pull request, #375: URL: https://github.com/apache/doris-spark-connector/pull/375
## What `RowBatch.getDateTime` treated every timezone-aware Arrow timestamp as a `TIMESTAMPTZ` instant in the JVM default zone. Doris `DATETIME` / `DATETIMEV2` store naive wall-clock digits. Some backends still emit Arrow `Timestamp` with a timezone (for example `+08:00`). The epoch instant plus that timezone is the original warehouse clock. Decoding in `ZoneId.systemDefault()` drops those digits (8 hours off when the JVM zone is GMT/UTC and Arrow is `+08:00`). This change: - `DATETIME` / `DATETIMEV2` with an Arrow timezone: decode in `ZoneId.of(arrowTz)` - `TIMESTAMPTZ`: keep the current instant decode in the JVM default zone - missing Arrow timezone: keep `getObject()` when it returns `LocalDateTime` `Timestamp.valueOf` and the Java 8 Instant path are unchanged. They still treat the recovered `LocalDateTime` as local in the Spark JVM zone. ## Why 26.1.0 is not enough https://github.com/apache/doris-spark-connector/pull/366 only special-cases a null Arrow timezone. Staging Doris still sends timezone-aware DATETIME, so 26.1.0 still prints the GMT face of the instant. Doris core https://github.com/apache/doris/pull/65823 emits naive Arrow DATETIME. That is the long-term fix. This connector change covers backends that still send a timezone on DATETIME. ## Test plan - [x] `RowBatchTest.testDatetimeTzVectorRecoversWallClock`: `DATETIME`/`DATETIMEV2` + `+08:00`, instant `2026-08-24T02:00:09Z` -> `Timestamp.valueOf("2026-08-24 10:00:09")` - [x] `RowBatchTest.testTimestampTzVector` still passes - [x] `RowBatchTest.testLongToLocalDateTimeUsesDeclaredUnit` still passes ``` cd spark-doris-connector JAVA_TOOL_OPTIONS='--add-opens=java.base/java.nio=ALL-UNNAMED' \ mvn -P spark-3.5 -pl spark-doris-connector-base test \ -Dtest=RowBatchTest#testDatetimeTzVectorRecoversWallClock,RowBatchTest#testTimestampTzVector,RowBatchTest#testLongToLocalDateTimeUsesDeclaredUnit ``` ## Related We reproduced this on Spark 3.5 + connector 25.2.0 / 26.1.0 against a Doris FE that reports `time_zone=UTC`, with Arrow `TimeStampMilliTZVector` timezone `+08:00`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
