924060929 opened a new pull request, #66710:
URL: https://github.com/apache/doris/pull/66710
[fix](fe) Keep same-file external splits on one backend
### What problem does this PR solve?
Issue Number: close #66601
Related PR: #66602
Problem Summary: External Parquet and ORC files can be divided into multiple
scan splits. Assigning those splits to different backends makes each backend
open the same remote file and independently read file-level metadata,
increasing remote I/O. Keep eligible splits from the same physical file on a
stable backend through file-key consistent hashing while preserving every split
as an independent scheduling unit. Explicitly limit affinity to native
Parquet/ORC ranges, preserve HDFS block locality and virtual/JNI connector
behavior, and keep affinity splits out of load redistribution.
### Release note
Improve external Parquet and ORC split placement by keeping ranges of the
same file on one backend.
### Check List (For Author)
- Test: Unit Test
- `FederationBackendPolicyTest` (19 tests)
- `./build.sh --fe` Maven reactor and Checkstyle passed; output assembly
returned non-zero after the successful reactor without an error message
- Behavior changed: Yes. Eligible ranges of one remote Parquet/ORC file are
assigned to one backend while remaining separate scan ranges.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
