Gabriel39 opened a new pull request, #69163:
URL: https://github.com/apache/doris/pull/69163
### What problem does this PR solve?
Backport #68934 (575c5ac6c0ec67dc8880b5fef6116f6f84d896e3) to branch-4.2.
Preserve nanosecond precision through Iceberg and Paimon reads, inserts and
CTAS, including arrays, maps and structs.
| Changed external type | Previous Doris mapping | New Doris mapping |
| --- | --- | --- |
| Iceberg `timestamp_ns` (without time zone) | Unsupported | `TIMESTAMP_NS` |
| Paimon `timestamp(7)`, `timestamp(8)`, `timestamp(9)` | `DATETIMEV2(6)` |
`TIMESTAMP_NS` |
Doris `TIMESTAMP_NS` writes as Iceberg `timestamp_ns` or Paimon
`timestamp(9)`. Iceberg nanosecond schemas require format version 3 and the
format V2 reader. Paimon high-precision reads select JNI to retain all
fractional digits.
Adapt the FE changes to the existing branch-4.2 datasource modules and
predicate APIs. Preserve branch-4.2's ORC invalid-DATE stripe safeguards when
grouping stripes by writer timezone, with a combined regression test. Keep the
source PR's nanosecond partition/default handling, retained-DATE metric guards,
strict schema-evolution overflow validation, and Parquet/ORC read/write
coverage. Support signed nanosecond transport in branch-4.2's legacy JNI
connector, covered by boundary, null and multi-batch tests. Paimon physical
metadata column assertions are omitted because that separate master feature is
absent from branch-4.2.
### Release note
Support reading and writing Iceberg `timestamp_ns` and Paimon
`timestamp(7/8/9)` as Doris `TIMESTAMP_NS`, preserving nanosecond precision.
### Validation
- Full FE and BE builds passed.
- FE Checkstyle and clang-format 16 passed for all affected files.
- 338 focused FE/Java tests passed.
- 632 focused BE tests passed under ASAN.
- 10 regression suites passed on a locally built FE/BE: Iceberg and Paimon
nanosecond round trips; Paimon timestamp types, MinIO and table properties;
HDFS Parquet groups 2/5/6; ORC exception files; TVF error columns.
- Regenerated branch-specific Paimon golden results through actual
execution. Local HDFS runs used a longer planning timeout after initial timeout
failures.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Regression test
- Behavior changed:
- [x] Yes. External unzoned nanosecond timestamps retain nine fractional
digits.
- Does this need documentation?
- [x] Yes. External type mapping documentation should include
`TIMESTAMP_NS`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
