yihua commented on code in PR #8856:
URL: https://github.com/apache/hudi/pull/8856#discussion_r1212191959
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -842,7 +842,8 @@ public static HoodieData<HoodieRecord>
convertFilesToBloomFilterRecords(HoodieEn
List<Pair<String, List<String>>> partitionToDeletedFilesList =
partitionToDeletedFiles.entrySet()
.stream().map(e -> Pair.of(e.getKey(),
e.getValue())).collect(Collectors.toList());
- int parallelism = Math.max(Math.min(partitionToDeletedFilesList.size(),
recordsGenerationParams.getBloomIndexParallelism()), 1);
+ int totalDeleteFileSize = partitionToDeletedFilesList.stream().mapToInt(p
-> p.getValue().size()).sum();
+ int parallelism = Math.max(Math.min(totalDeleteFileSize,
recordsGenerationParams.getBloomIndexParallelism()), 1);
Review Comment:
The number of entries in the list (`partitionToDeletedFilesList`,
`partitionToAppendedFiles`) is the number of table partitions. Parallelizing
the list to RDD with a higher parallelism does not have any impact on the
execution in Spark since one executor still gets a pair of all info within a
table partition, unless the DAG is rewritten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
