chia7712 commented on code in PR #18012:
URL: https://github.com/apache/kafka/pull/18012#discussion_r1909370800
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -232,38 +232,48 @@ private boolean canConvertToRelativeOffset(long offset)
throws IOException {
* It is assumed this method is being called from within a lock, it is not
thread-safe otherwise.
*
* @param largestOffset The last offset in the message set
- * @param largestTimestampMs The largest timestamp in the message set.
- * @param shallowOffsetOfMaxTimestamp The last offset of earliest batch
with max timestamp in the messages to append.
- * @param records The log entries to append.
+ * @param records The log entries to append.
* @throws LogSegmentOffsetOverflowException if the largest offset causes
index offset overflow
*/
public void append(long largestOffset,
- long largestTimestampMs,
- long shallowOffsetOfMaxTimestamp,
MemoryRecords records) throws IOException {
if (records.sizeInBytes() > 0) {
- LOGGER.trace("Inserting {} bytes at end offset {} at position {}
with largest timestamp {} at offset {}",
- records.sizeInBytes(), largestOffset, log.sizeInBytes(),
largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes());
int physicalPosition = log.sizeInBytes();
- if (physicalPosition == 0)
- rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+ boolean updateRollingBasedTimestamp = physicalPosition == 0;
ensureOffsetInRange(largestOffset);
// append the messages
long appendedBytes = log.append(records);
LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes,
log.file(), largestOffset);
- // Update the in memory max timestamp and corresponding offset.
- if (largestTimestampMs > maxTimestampSoFar()) {
- maxTimestampAndOffsetSoFar = new
TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
- }
- // append an entry to the index (if needed)
- if (bytesSinceLastIndexEntry > indexIntervalBytes) {
- offsetIndex().append(largestOffset, physicalPosition);
- timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
- bytesSinceLastIndexEntry = 0;
+
+ long recordsLargestTimestampMs = RecordBatch.NO_TIMESTAMP;
+ for (RecordBatch batch : records.batches()) {
+ long batchMaxTimestamp = batch.maxTimestamp();
+ long batchLastOffset = batch.lastOffset();
+ recordsLargestTimestampMs =
Math.max(recordsLargestTimestampMs, batchMaxTimestamp);
+ boolean updateTimeIndex = false;
+ if (batchMaxTimestamp > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new
TimestampOffset(batchMaxTimestamp, batchLastOffset);
+ updateTimeIndex = true;
+ }
+
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batchLastOffset, physicalPosition);
+
+ // max timestamp may not be monotonic, so we need to check
it to avoid the time index append error
+ if (updateTimeIndex)
timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
Review Comment:
The purpose of using updateTimeIndex is to minimize unnecessary calls to
maybeAppend by ensuring that it's only invoked when a larger timestamp is
encountered. That can avoid requiring unnecessary locking, but I think maybe
that is unnecessary optimization. We can totally remove `updateTimeIndex` to
simplify code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
