FrankYang0529 commented on code in PR #18012:
URL: https://github.com/apache/kafka/pull/18012#discussion_r1908982266
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -253,17 +253,33 @@ public void append(long largestOffset,
// append the messages
long appendedBytes = log.append(records);
LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes,
log.file(), largestOffset);
- // Update the in memory max timestamp and corresponding offset.
- if (largestTimestampMs > maxTimestampSoFar()) {
- maxTimestampAndOffsetSoFar = new
TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
- }
- // append an entry to the index (if needed)
- if (bytesSinceLastIndexEntry > indexIntervalBytes) {
- offsetIndex().append(largestOffset, physicalPosition);
- timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
- bytesSinceLastIndexEntry = 0;
+
+ long batchMaxTimestamp = RecordBatch.NO_TIMESTAMP;
+ long batchShallowOffsetOfMaxTimestamp = -1L;
+ // append an entry to the index at batches level (if needed)
+ for (RecordBatch batch : records.batches()) {
+ if (batch.maxTimestamp() > batchMaxTimestamp) {
+ batchMaxTimestamp = batch.maxTimestamp();
+ batchShallowOffsetOfMaxTimestamp = batch.lastOffset();
+ }
+
+ // Update the in memory max timestamp and corresponding offset.
+ if (batchMaxTimestamp > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new
TimestampOffset(batchMaxTimestamp, batchShallowOffsetOfMaxTimestamp);
+ }
+
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), physicalPosition);
+
+ // max timestamp may not be monotonic, so we need to check
it to avoid the time index append error
+ if (batchMaxTimestamp >= timeIndex().lastEntry().timestamp)
+ timeIndex().maybeAppend(batchMaxTimestamp,
shallowOffsetOfMaxTimestampSoFar());
Review Comment:
Yes, updated it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
