Gabriel39 opened a new pull request, #69320:
URL: https://github.com/apache/doris/pull/69320
### What problem does this PR solve?
Traditional file loads with source/destination tuples always select
FileScanner V1. Add `LoadTableReader` around the existing V2 source reader so
supported loads can use FileScannerV2 without FE or Thrift changes.
The adapter materializes source defaults and path columns, evaluates source
pre-filters and destination expressions, checks strict-mode conversion and
nullability, and preserves flexible partial-update skip bitmaps and
sequence-column marks. Scanner continues to evaluate destination conjuncts
before ordinary output projections. Source readers receive no destination
predicate, aggregate pushdown, or condition-cache snapshot.
CSV/JSON readers reject malformed load records with row rollback and error
accounting. JSON reports field-presence masks separately from NULL values so
missing defaults and partial updates retain their meaning. Fully rejected
batches do not end a split.
Supported load formats select V2 independently of the query session option:
the Thrift default does not distinguish an omitted option from false in load
plans. Query selection remains unchanged. Avro and Arrow IPC stream loads keep
their existing path; the current V2 Arrow implementation only supports
remote-Doris Flight scans. Load ranges keep their FE split boundaries.
This PR targets `branch-4.2` and is independent of #69312.
### Validation
- Passed 13 targeted tests in a local ASAN/UBSAN harness, linking rebuilt
changed translation units against existing project libraries. Coverage includes
strict/non-strict conversion, pre-filter order, explicit NULL versus missing
defaults, non-nullability, flexible updates and sequence marks, rejected-batch
EOF, scanner selection, and real CSV/JSON readers through the adapter.
- Recompiled the changed BE implementation units and syntax-checked the
affected existing scanner, CSV, and JSON test suites.
- Passed clang-format 16 checks for all 17 affected C++ source/header files
and `git diff --check`.
- Added parser unit tests and a Stream Load regression covering
target-column WHERE, rejected/unselected counts, JSON defaults, and flexible
partial updates. Full BE build and cluster regression execution remain for CI;
they were not run locally.
### Release note
Route supported traditional file loads through FileScannerV2 with a
table-reader adapter for load conversion and validation.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Supported traditional loads select V2 independently of the
query-only session switch.
- Does this need documentation?
- [x] Yes. Updated the in-repository FileScannerV2 design document.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
