Gabriel39 opened a new pull request, #69668:
URL: https://github.com/apache/doris/pull/69668
### What problem does this PR solve?
Arrow Stream Load still selects FileScanner V1, and CSV/TEXT/JSON schema
discovery still constructs legacy readers even when the subsequent scan uses V2.
Add a native Arrow IPC FileReader to the V2 load path. It decodes record
batches incrementally, supports concatenated streams and compressed IPC, uses
tracked allocations, and retains load expressions, defaults, strict-mode
checks, and WHERE filtering. Source timestamp units and timezones are preserved
for Doris SerDe rounding, including nested values.
Route CSV, TEXT, and JSON schema discovery through V2 readers and the shared
schema entry point, alongside the existing native Parquet/ORC readers. Preserve
header handling, nullable string inference, JSON root/path options, and
independent sampling of HTTP Stream Load data. Also avoid consuming a rooted
JSON array's ondemand cursor before jsonpaths materialization.
### Release note
Arrow Stream Load and CSV/TEXT/JSON schema discovery now use FileScanner V2
readers.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- ASAN BE build and 172 focused unit tests passed, including scanner
routing, native schema inference, Arrow IPC loading, timestamp rounding, and
existing reader coverage.
- Six regression suites passed against a locally built FE/ASAN BE:
Arrow Stream Load, V2 HTTP schema discovery, and existing JSON, compressed CSV,
CSV-with-names, and CSV-with-names-and-types suites. Results were compared with
expected output.
- clang-format 16, changed-line clang-tidy, and build hygiene checks
passed.
- Behavior changed:
- [x] Yes. Arrow loads and text/JSON schema inference use native V2
readers.
- Does this need documentation?
- [x] No. Existing load interfaces and options are retained.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
