yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2214352748
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1046,40 +1048,14 @@ private static <T> ClosableIterator<BufferedRecord<T>>
getLogRecords(List<String
.withMetaClient(datasetMetaClient)
.withAllowInflightInstants(true)
.withRecordBuffer(recordBuffer)
- .build();
- return new CloseableLogRecordsIterator<>(mergedLogRecordReader,
recordBuffer);
+ .build()) {
+ // initializes the record buffer with the log records
+ return recordBuffer.getLogRecordIterator();
Review Comment:
This is an existing issue, which we should track separately.
`recordBuffer.getLogRecordIterator()` invokes `ExternalSpillableMap#values`
which puts all in-memory and disk-spilled records to a in-memory list which can
cause OOM.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -394,4 +372,29 @@ static Comparable getOrderingValue(HoodieReaderContext
readerContext,
? DEFAULT_ORDERING_VALUE
:
readerContext.convertValueToEngineType(deleteRecord.getOrderingValue());
}
+
+ private static class LogRecordIterator<T> implements
ClosableIterator<BufferedRecord<T>> {
+ private final FileGroupRecordBuffer<T> fileGroupRecordBuffer;
+ private final Iterator<BufferedRecord<T>> logRecordIterator;
+
+ private LogRecordIterator(FileGroupRecordBuffer<T> fileGroupRecordBuffer) {
+ this.fileGroupRecordBuffer = fileGroupRecordBuffer;
+ this.logRecordIterator =
fileGroupRecordBuffer.records.values().iterator();
Review Comment:
Regarding the OOM comment I have on `HoodieTableMetadataUtil`, it can be
fixed by using `fileGroupRecordBuffer.records.iterator()` instead of
`fileGroupRecordBuffer.records.values().iterator()`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/HoodieLogCompactionPlanGenerator.java:
##########
@@ -55,7 +54,6 @@ public HoodieLogCompactionPlanGenerator(HoodieTable table,
HoodieEngineContext e
super(table, engineContext, writeConfig, executor);
this.compactionStrategy = HoodieCompactionStrategy.newBuilder()
.setStrategyParams(getStrategyParams())
- .setCompactorClassName(LogCompactionExecutionHelper.class.getName())
Review Comment:
This field is used to determine whether log compaction execution should be
used by existing releases. So we should just backfill the same class name for
backwards compatibility.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java:
##########
@@ -332,7 +332,7 @@ protected static Option<IndexedRecord>
toAvroRecord(HoodieRecord record, Schema
try {
return record.toIndexedRecord(writerSchema,
props).map(HoodieAvroIndexedRecord::getData);
} catch (IOException e) {
- LOG.error("Fail to get indexRecord from " + record, e);
+ LOG.error("Failed to convert to IndexedRecord", e);
Review Comment:
Does this cause any issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
