danny0405 commented on code in PR #13449:
URL: https://github.com/apache/hudi/pull/13449#discussion_r2155925595
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -546,12 +557,78 @@ public List<WriteStatus> close() {
status.getStat().setFileSizeInBytes(logFileSize);
}
+ // generate Secondary index stats if streaming is enabled.
+ if (!isSecondaryIndexStreamingDisabled()) {
+ // Adds secondary index only for the last log file write status. We do
not need to add secondary index stats
+ // for every log file written as part of the append handle write. The
last write status would update the
+ // secondary index considering all the log files.
+
trackMetadataIndexStatsForStreamingMetadataWrites(fileSliceOpt.or(this::getFileSlice),
statuses.stream().map(status ->
status.getStat().getPath()).collect(Collectors.toList()),
+ statuses.get(statuses.size() - 1));
+ }
+
return statuses;
} catch (IOException e) {
throw new HoodieUpsertException("Failed to close UpdateHandle", e);
}
}
+ private void
trackMetadataIndexStatsForStreamingMetadataWrites(Option<FileSlice>
fileSliceOpt, List<String> newLogFiles, WriteStatus status) {
Review Comment:
move all the index stats related changes into its corresbonding utility
classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
