Gabriel39 opened a new pull request, #68924:
URL: https://github.com/apache/doris/pull/68924
### What problem does this PR solve?
Paimon's default legacy ORC encoding for `TIMESTAMP_LTZ` depends on the
writer and reader JVM time zones. For example, an instant written in a UTC JVM
and read in an Asia/Shanghai JVM shifts by eight hours, even when both engines
agree on the SQL session time zone. Doris currently inherits this legacy
default when creating Paimon tables.
Persist `orc.timestamp-ltz.legacy.type=false` when Doris creates a Paimon
table. Explicit table settings take precedence over the Paimon catalog's
`table-default.orc.timestamp-ltz.legacy.type`, and both take precedence over
the new default. Persist the option for all new tables so adding an LTZ column
or subsequently selecting ORC does not silently enable legacy encoding.
Existing tables retain their encoding settings. This change does **not**
repair historical legacy files or change tables created by Spark/Flink. Modern
and legacy LTZ files share the same ORC physical type, and LTZ-only legacy
files may omit the writer time zone, so an automatic reinterpretation would be
unsafe. Existing legacy data needs a controlled rewrite into a table with the
modern encoding; changing the property alone is not a migration.
### Release note
New Paimon tables created through Doris default to portable ORC
`TIMESTAMP_LTZ` encoding. Explicit table and catalog defaults remain supported,
and existing table options are unchanged.
### Validation
- Compiled the changed production and test classes and ran
`PaimonMetadataOpsTest` with JUnit using an existing branch-4.1 FE dependency
classpath: 28 tests passed. Before the fix, the new default-encoding test
failed because the persisted option was absent.
- Repository FE Checkstyle (`mvn -pl fe-core checkstyle:check`): passed with
zero violations.
- Added coverage for ORC/Parquet tables, LTZ/NTZ schema types, explicit
legacy/modern options, catalog defaults, table overrides, and preservation of
existing legacy options through `IF NOT EXISTS` and unrelated `ALTER`.
- Real Paimon 1.4.2 ORC `FormatWriter`/`FormatReader` probe with
`orc.timestamp-ltz.legacy.type=false`: separate JVMs using
`-Duser.timezone=UTC`, `Asia/Shanghai`, and `America/Los_Angeles`, all nine
writer/reader combinations. Ten synthetic timestamps covering winter/summer,
DST boundaries, microseconds, epoch zero, a negative epoch and 2038 produced
180/180 LTZ/NTZ value assertions and 9/9 row-count assertions passing.
- Validation limits: no full Doris/Spark/Flink end-to-end deployment was
run. A separate pre-epoch subsecond edge (`-1us` and `-1001us`) fails with the
same +1-second result under both legacy and modern SDK encoding in UTC; that
existing SDK issue is outside this time-zone-default change.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (real Paimon ORC files in separate writer/reader JVMs)
- Behavior changed:
- [x] Yes. New-table default only, as described above.
- Does this need documentation?
- [x] Yes. Document the new default and the requirement to rewrite
historical legacy data before opting into modern encoding.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
