Gabriel39 opened a new pull request, #66008:
URL: https://github.com/apache/doris/pull/66008
### What problem does this PR solve?
Problem Summary:
FileScannerV2 did not accept `FORMAT_JNI`, `FORMAT_WAL`, or `FORMAT_AVRO`,
so these scans remained on or depended on the legacy scanner path.
This change:
- routes supported `FORMAT_JNI` scans through FileScannerV2, including
compatible Paimon JNI/native split shapes;
- adds an independent V2 WAL reader with unique-column-id mapping,
block-version validation, checksum-backed WAL file reads, projection, and
filtering;
- adds an independent V2 Avro JNI reader and Java scanner with split reads,
projection, HDFS/S3 configuration, nested/union values, and schema discovery;
- keeps Avro away from table formats that require delete/merge wrappers;
- does not include or call the legacy WAL or Avro reader implementations.
### Release note
FileScannerV2 now supports JNI scans, WAL files, and Avro files.
### Check List (For Author)
- Test
- [x] Unit Test
Verified with:
- `./run-be-ut.sh --run
--filter='FileScannerV2Test.*:WalReaderV2Test.*:AvroJniReaderV2Test.*' -j 8`
(25 tests passed)
- `mvn -pl be-java-extensions/avro-scanner -am test
-Dtest=AvroJniScannerTest,AvroColumnValueTest
-Dsurefire.failIfNoSpecifiedTests=false -Dmaven.build.cache.enabled=false` (5
tests passed)
- `mvn -pl be-java-extensions/avro-scanner -am package -DskipTests
-Dmaven.build.cache.enabled=false`
- Behavior changed:
- [x] Yes. Eligible JNI, WAL, and Avro scans use FileScannerV2 when
`enable_file_scanner_v2` is enabled.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
