github-actions[bot] commented on code in PR #66710:
URL: https://github.com/apache/doris/pull/66710#discussion_r3780701086
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:
##########
@@ -472,7 +485,31 @@ public void createScanRangeLocations() throws
UserException {
}
}
- private TScanRangeLocations splitToScanRange(
+ protected boolean supportsPerRangeFileAffinity() {
+ return false;
+ }
+
+ @Override
+ public List<List<Integer>> splitScanRangeParamsByInstance(
+ List<TScanRangeParams> scanRangeParams, int expectedInstanceNum) {
+ return AffinityAwareInstanceSplitter.split(scanRangeParams,
expectedInstanceNum,
+ param ->
Optional.ofNullable(scanRangeToFileAffinityKey.get(param.getScanRange())),
+ this::getScanRangeSize);
+ }
+
+ private long getScanRangeSize(TScanRangeParams param) {
+ long size = 0;
+ for (TFileRangeDesc range :
param.getScanRange().getExtScanRange().getFileScanRange().getRanges()) {
Review Comment:
`TFileScanRange.ranges` is optional and unset for every batch-mode outer
range constructed with only `split_source`. This is distinct from the earlier
lazy-affinity wiring thread: `AffinityAwareInstanceSplitter` invokes this size
callback before it checks that no affinity key exists, so legacy non-serial and
Nereids ordinary/local-shuffle planning dereference the null list here and
abort large or streaming external scans before the BE can fetch splits. Please
short-circuit to the round-robin path before evaluating weights (or explicitly
handle split-source-only ranges), and add production-shaped legacy/Nereids
regressions with `split_source` set and `ranges` unset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
