TheR1sing3un commented on code in PR #13066:
URL: https://github.com/apache/hudi/pull/13066#discussion_r2025943342
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/DataSourceInternalWriterHelper.java:
##########
@@ -85,14 +97,49 @@ public void commit(List<WriteStatus> writeStatuses) {
try {
List<HoodieWriteStat> writeStatList =
writeStatuses.stream().map(WriteStatus::getStat).collect(Collectors.toList());
writeClient.commitStats(instantTime, writeStatList,
Option.of(extraMetadata),
- CommitUtils.getCommitActionType(operationType,
metaClient.getTableType()));
+ CommitUtils.getCommitActionType(operationType,
metaClient.getTableType()), getReplacedFileIds(writeStatuses), Option.empty());
} catch (Exception ioe) {
throw new HoodieException(ioe.getMessage(), ioe);
} finally {
writeClient.close();
}
}
+ private Map<String, List<String>> getReplacedFileIds(List<WriteStatus>
writeStatuses) {
Review Comment:
> Sorry, I still think keeping `getReplacedFileIds` in each executor is more
cleaner and reasonable.
Your suggestion is reasonable. However, since the current executor is only
used to do the bulk_insert pre-preparation, and then the rest of the writes and
commits are handed over to `DataSourceInternalWriterHelper`,
`executor::getReplaceFileIds` are not called. Because executor is not
responsible for committing, the `getReplaceFileId` method is not required. My
original intention was just to converge the actual execution and commit code
paths of `bulk_insert`/`insert_overwrite`/`insert_overwrite_table` together,
instead of this: Some use `HoodieDatasetBulkInsertHelper`, some
`DataSourceInternalWriterHelper`.
So do you have any good suggestions? Looking forward to your reply~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
