manesioz opened a new issue, #376: URL: https://github.com/apache/doris-spark-connector/issues/376
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris-spark-connector/issues?q=is%3Aissue) and found no similar issues. ### Version spark-doris-connector-spark-3.5 25.2.0 and master (including the DATETIME / TIMESTAMPTZ Arrow change in #366) ### What's Wrong? Doris `DATETIME` / `DATETIMEV2` store a clock with no time zone. Some Doris backends still send those columns as Arrow `TimeStamp*TZVector`. The Arrow type has a timezone string (for example `+08:00` or `UTC+8`). The cell is a UTC epoch. `RowBatch.getDateTime` treats every Arrow timestamp that has a timezone the same way. It converts the epoch to `LocalDateTime` in `ZoneId.systemDefault()`. Then `Timestamp.valueOf` uses that `LocalDateTime` as local time in the Spark JVM. If the JVM zone is UTC (or GMT) and the Arrow timezone is `+08:00`, Spark returns the UTC face of the epoch. That is not the clock stored in Doris. Example: - Doris `DATETIME` value: `2026-08-24 10:00:09` - Arrow: epoch `2026-08-24T02:00:09Z`, `ArrowType.Timestamp.getTimezone()` = `+08:00` - Spark on a UTC JVM: `2026-08-24 02:00:09` (JSON `2026-08-24T02:00:09.000Z`) `TIMESTAMPTZ` is an instant. The current decode is correct for that type. `DATETIME` and `TIMESTAMPTZ` share one `switch` case and the same `getDateTime` path, so the connector cannot tell them apart. #366 only special-cases a **null** Arrow timezone (`getObject()`). It does not fix `DATETIME` when the Arrow timezone is set. https://github.com/apache/doris/pull/65823 makes `DATETIME` a naive Arrow timestamp. Backends that still send a timezone on `DATETIME` need a connector change. ### What You Expected? Spark should return the clock stored in the `DATETIME` column (`2026-08-24 10:00:09` in the example). For a labeled Arrow timestamp: - `DATETIME` / `DATETIMEV2`: `LocalDateTime` in `ZoneId.of(arrowTimezone)`, then the existing `Timestamp.valueOf` / Java 8 `Instant` write - `TIMESTAMPTZ`: keep #366 (`getDateTime` with `ZoneId.systemDefault()`) ### How to Reproduce? 1. Use a Doris backend that still encodes `DATETIME` as `TimeStampMilliTZVector` (or micro/sec TZ) with a non-null timezone on the Arrow type. 2. Store `DATETIME` `2026-08-24 10:00:09` (no zone on the column). 3. Read the table with Spark Doris connector. Spark JVM time zone UTC. 4. Compare Spark `java.sql.Timestamp` (or JSON) with a JDBC/`SELECT` of the same row. You can also build an Arrow batch in `RowBatchTest` with the same epoch on two columns, schema `TIMESTAMPTZ` vs `DATETIME`. The two Spark values must differ. ### Anything Else? Related: - https://github.com/apache/doris/pull/38215 (timezone-aware Arrow for datetime) - https://github.com/apache/doris-spark-connector/pull/366 (null timezone vs labeled timestamp, `TIMESTAMPTZ`) - https://github.com/apache/doris/pull/65823 (naive Arrow `DATETIME`) This is not issue #350 (pre-1970 `Instant` range). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
