924060929 commented on code in PR #70664:
URL: https://github.com/apache/doris/pull/70664#discussion_r4624591663
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:
##########
@@ -543,6 +545,57 @@ private TScanRangeLocations splitToScanRange(
return curLocations;
}
+ protected boolean supportsBeSplitRefinement(FileSplit split) {
+ return true;
+ }
+
+ protected long coarseSplitSizeForBeRefinement() {
+ return isCoarseSplitPlanningEnabled(sessionVariable) &&
!wouldRunSerialOnBe()
Review Comment:
[P2] Preserve scan parallelism across BEs. In synchronous Hive planning with
two eligible BEs and round-robin assignment, one 400 MiB Parquet file currently
produces two ranges at the default 256 MiB target. At 512 MiB it produces one
range, so all physical children run on the same BE. Can we account for
available BEs and expected range count before coarsening, and add a multi-BE
test that checks placement?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:
##########
@@ -1948,6 +1948,7 @@ public List<Split> getSplits(int numBackends) throws
UserException {
.requiredPartitions(requiredPartitions)
.partitionsPrunedToEmpty(partitionsPrunedToEmpty)
.countPushdown(countPushdown)
+ .coarseSplitSize(coarseSplitSizeForBeRefinement())
Review Comment:
[P2] Preserve Hive `TABLESAMPLE` granularity. Hive samples the ranges
returned by the connector. With `file_split_size=16 MiB`, a 4% sample of one
400 MiB Parquet file selects 16 MiB under the existing policy. The 512 MiB
coarse target produces one range and the same sample selects the whole file.
Can we use `applySample` to keep legacy FE sizing for sampled scans? The BE can
still refine the selected ranges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]