wombatu-kun commented on code in PR #10917:
URL: https://github.com/apache/hudi/pull/10917#discussion_r1536992371
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -1096,11 +1104,37 @@ private Set<String> getDropPartitionColNames() {
return new HashSet<>(Arrays.asList(partitionFields.get()));
}
- public void runMerge(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
+ public Iterator<List<WriteStatus>> handleUpdate(String instantTime, String
partitionPath, String fileId,
+ Map<String, HoodieRecord<T>>
keyToNewRecords, HoodieBaseFile oldDataFile) throws IOException {
+ HoodieMergeHandle upsertHandle = getUpdateHandle(instantTime,
partitionPath, fileId, keyToNewRecords, oldDataFile);
+ return updateWithUpsertHandle(upsertHandle, instantTime, fileId);
+ }
+
+ public Iterator<List<WriteStatus>>
updateWithUpsertHandle(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
+ runMerge(upsertHandle, instantTime, fileId);
+ return upsertHandle.getWriteStatusesAsIterator();
+ }
+
+ protected void runMerge(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String
instantTime, String fileId) throws IOException {
if (upsertHandle.getOldFilePath() == null) {
throw new HoodieUpsertException("Error in finding the old file path at
commit " + instantTime + " for fileId: " + fileId);
} else {
HoodieMergeHelper.newInstance().runMerge(this, upsertHandle);
}
}
+
+ protected HoodieMergeHandle getUpdateHandle(String instantTime, String
partitionPath, String fileId,
Review Comment:
Yes, this `handleUpdate` method belongs to `HoodieCompactionHandler`
interface, but `HoodieTable` is the only place where this code can be reused
for all engine types without duplications.
I don't think it's possible to merge `runMerge`, `updateWithUpsertHandle`
and `handleUpdate` into one methods. I tried and had these problems:
- `updateWithUpsertHandle` is used not only in compaction process, but in
XXXCommitActionExecutors too;
- `runMerge` code is the same for Java and Flink engines, but it is
different for Spark, so it must be separated from `handleUpdate` and overriden;
- also it's impossible to move such realization of `handleUpdate` method to
`HoodieCompactionHandler` as default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
