924060929 commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3801151613
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -320,8 +322,17 @@ private void
getFileSplitByPartitions(HiveExternalMetaCache cache, List<HivePart
}
} else {
boolean withCache = Config.max_external_file_cache_num > 0;
- fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
- directoryLister, hmsTable);
+ HiveFileScanTaskCacheKey cacheKey = new HiveFileScanTaskCacheKey(
+ hmsTable.getCatalog().getId(), hmsTable.getId(),
partitions);
+ try {
+ fileCaches = getOrLoadExternalScanTasks(cacheKey,
Review Comment:
Resolved on the current head. Hive bypasses statement caching for batch mode
and disabled global file cache (`isBatchMode || !withCache`), Hudi bypasses for
batch partitions (`useStatementCache=false`), and Iceberg's manifest-cache
planner streams when split size is explicit, batch mode, or count pushdown is
active. All three were covered by tests in the batch/explicit-size round.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -380,29 +384,80 @@ private List<HivePartition>
getPrunedPartitions(HoodieTableMetaClient metaClient
private List<Split> getIncrementalSplits() {
long startTime = System.currentTimeMillis();
- if (canUseNativeReader()) {
- List<Split> splits = incrementalRelation.collectSplits();
- noLogsSplitNum.addAndGet(splits.size());
+ try {
+ HudiIncrementalScanTaskCacheKey cacheKey = new
HudiIncrementalScanTaskCacheKey(
Review Comment:
Resolved on the current head. `getIncrementalSplits()` no longer uses the
statement cache at all: incremental planning always consumes the directly
planned splits, so no key can alias historical instants.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -380,29 +384,80 @@ private List<HivePartition>
getPrunedPartitions(HoodieTableMetaClient metaClient
private List<Split> getIncrementalSplits() {
long startTime = System.currentTimeMillis();
- if (canUseNativeReader()) {
- List<Split> splits = incrementalRelation.collectSplits();
- noLogsSplitNum.addAndGet(splits.size());
+ try {
+ HudiIncrementalScanTaskCacheKey cacheKey = new
HudiIncrementalScanTaskCacheKey(
+ hmsTable.getCatalog().getId(), hmsTable.getId(),
+ incrementalRelation.getStartTs(),
incrementalRelation.getEndTs(),
+ canUseNativeReader(),
incrementalRelation.getHoodieParams());
+ List<HudiSplit> plannedSplits =
getOrLoadExternalScanTasks(cacheKey, () -> {
+ if (canUseNativeReader()) {
+ return incrementalRelation.collectSplits().stream()
+ .map(split -> {
+ Preconditions.checkState(split instanceof
HudiSplit,
+ "Hudi COW incremental relation must
produce HudiSplit");
+ return (HudiSplit) split;
+ })
+ .collect(Collectors.toList());
+ }
+ Option<String[]> partitionColumns =
hudiClient.getTableConfig().getPartitionFields();
+ List<String> partitionNames = partitionColumns.isPresent()
+ ? Arrays.asList(partitionColumns.get()) :
Collections.emptyList();
+ return incrementalRelation.collectFileSlices().stream()
+ .map(fileSlice -> generateHudiSplit(fileSlice,
+ HudiPartitionUtils.parsePartitionValues(
+ partitionNames,
fileSlice.getPartitionPath()),
+ incrementalRelation.getEndTs()))
+ .collect(Collectors.toList());
+ });
+ List<Split> splits = plannedSplits.stream()
Review Comment:
Resolved together with the above. With incremental planning uncached there
is no retained planned list and no per-file deep copy beyond the single direct
plan.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -1367,9 +1461,11 @@ private List<Split> doGetSystemTableSplits() throws
UserException {
List<Split> splits = new ArrayList<>();
TableScan scan = createTableScan();
long startTime = System.currentTimeMillis();
- try (CloseableIterable<FileScanTask> fileScanTasks = scan.planFiles())
{
+ try {
+ List<FileScanTask> fileScanTasks = getOrLoadExternalScanTasks(
Review Comment:
Addressed on the current head. Metadata-table planning goes through the
serialized-task cache under the statement-wide ICEBERG_SERIALIZED_BYTES
cumulative budget; on overflow the plan is consumed directly and nothing is
retained in StatementContext. Covered by testOversizedIcebergPlanIsNotRetained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
