Gabriel39 opened a new pull request, #68649:
URL: https://github.com/apache/doris/pull/68649
### What problem does this PR solve?
Related issue: #67366
Related PR: #67449
On branch-4.1, Arrow Flight SQL returns `0000-01-02` for a stored DATE of
`0000-01-01`. Doris follows the MySQL calendar, which excludes year zero's leap
day, while Arrow DATE uses the proleptic Gregorian calendar. Subtracting the
Unix epoch day number directly shifts dates in `0000-01-01` through
`0000-02-28` by one day, including elements of `ARRAY<DATE>`.
Adapt the DATE portion of #67449 to branch-4.1, with credit to its author.
Convert between the calendars at Arrow and ORC write boundaries and
Arrow/default Parquet/ORC read boundaries. Use the same conversion for ORC
statistics so pushed-down MIN/MAX agrees with row decoding. Reject
unrepresentable external dates and ignore undefined payloads in null Arrow
slots.
The two DATE regression suites and their expected results are copied
unchanged from #67449. Add its DATE unit coverage plus an explicit null-payload
test and an exhaustive comparison against the civil calendar for all 3,652,424
representable dates.
Scope: DATE only. This does not include the DATETIME/Iceberg changes in
#67449 or the independent legacy file scanner conversions used when
`enable_file_scanner_v2=false`.
### Validation
- Baseline: 9 of the original 10 DATE unit tests fail; the unchanged Flight
regression reproduces the one-day shift.
- Fixed: 26 DATE SerDe/ORC reader unit tests pass under ASAN, including the
exhaustive calendar check. These compile the modified branch-4.1 sources
against cached test dependencies.
- The unchanged `test_date_year_zero` Flight regression passes locally,
including nullable and array values.
- `test_outfile_date_year_zero` passes for Parquet and ORC using the
original rows and golden assertions, with S3 transport replaced by local
outfile/TVF only for the local run. The committed suite retains S3 transport
for CI.
- Local integration runs use the same fix rebuilt into an existing
ASAN/UBSAN BE build at `5bcb996a5ff`, with an existing FE. A full branch-4.1
build and the original S3 run remain for CI.
- clang-format 16 passes for all five affected C++ files. Source whitespace
checks pass; generated golden files retain their conventional trailing blank
line from #67449.
### Release note
Fix year-zero DATE values being returned one day late through Arrow Flight
SQL, and align DATE encoding and decoding in the default Parquet/ORC paths.
Existing files written with the incorrect year-zero encoding cannot be
distinguished automatically from correctly encoded files. Values in that narrow
date range should be re-exported from the original data after upgrading. The
proleptic-only `0000-02-29` remains unrepresentable as a Doris DATE.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes, external DATE ordinals use the correct calendar; internal DATE
arithmetic is unchanged.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
