wombatu-kun commented on code in PR #10917:
URL: https://github.com/apache/hudi/pull/10917#discussion_r1536992371
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -1096,11 +1104,37 @@ private Set<String> getDropPartitionColNames() {
return new HashSet<>(Arrays.asList(partitionFields.get()));
}
- public void runMerge(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
+ public Iterator<List<WriteStatus>> handleUpdate(String instantTime, String
partitionPath, String fileId,
+ Map<String, HoodieRecord<T>>
keyToNewRecords, HoodieBaseFile oldDataFile) throws IOException {
+ HoodieMergeHandle upsertHandle = getUpdateHandle(instantTime,
partitionPath, fileId, keyToNewRecords, oldDataFile);
+ return updateWithUpsertHandle(upsertHandle, instantTime, fileId);
+ }
+
+ public Iterator<List<WriteStatus>>
updateWithUpsertHandle(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
+ runMerge(upsertHandle, instantTime, fileId);
+ return upsertHandle.getWriteStatusesAsIterator();
+ }
+
+ protected void runMerge(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
if (upsertHandle.getOldFilePath() == null) {
throw new HoodieUpsertException("Error in finding the old file path at
commit " + instantTime + " for fileId: " + fileId);
} else {
HoodieMergeHelper.newInstance().runMerge(this, upsertHandle);
}
}
+
+ protected HoodieMergeHandle getUpdateHandle(String instantTime, String
partitionPath, String fileId,
Review Comment:
Yes, this handleUpdate method belongs to HoodieCompactionHandler inteface,
but HoodieTable is the only place where this code can be reused for all engine
types without duplications.
I don't think it's possible to merge runMerge, updateWithUpsertHandle and
handleUpdate into one methods. I tried and had these problems:
- updateWithUpsertHandle is used not only in compaction process, but in
CommitActionExecutors too;
- runMerge code is the same for Java and Flink engines, but it is different
for Spark, so it must be separated from handleUpdate and overriden;
- also it's impossible to move such realization of handleUpdate method to
HoodieCompactionHandler as default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
