[
https://issues.apache.org/jira/browse/KAFKA-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816120#comment-16816120
]
ASF GitHub Bot commented on KAFKA-7866:
---------------------------------------
hachikuji commented on pull request #6570: KAFKA-7866; Ensure no duplicate
offsets after txn index append failure
URL: https://github.com/apache/kafka/pull/6570
This patch fixes a bug in the append logic which can cause duplicate offsets
to be appended to the log when the append to the transaction index fails.
Rather than incrementing the log end offset after the index append, we do it
immediately after the records are written to the log. If the index append later
fails, we do two things:
1) We ensure that the last stable offset cannot advance. This guarantees
that the aborted data will not be returned to the user until the transaction
index contains the corresponding entry.
2) We skip updating the end offset of the producer state. When recovering
the log, we will have to reprocess the log and write the index entries.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Duplicate offsets after transaction index append failure
> --------------------------------------------------------
>
> Key: KAFKA-7866
> URL: https://issues.apache.org/jira/browse/KAFKA-7866
> Project: Kafka
> Issue Type: Bug
> Reporter: Jason Gustafson
> Priority: Major
>
> We have encountered a situation in which an ABORT marker was written
> successfully to the log, but failed to be written to the transaction index.
> This prevented the log end offset from being incremented. This resulted in
> duplicate offsets when the next append was attempted. The broker was using
> JBOD and we would normally expect IOExceptions to cause the log directory to
> be failed. That did not seem to happen here and the duplicates continued for
> several hours.
> Unfortunately, we are not sure what the cause of the failure was.
> Significantly, the first duplicate was also the first ABORT marker in the
> log. Unlike the offset and timestamp index, the transaction index is created
> on demand after the first aborted transction. It is likely that the attempt
> to create and open the transaction index failed. There is some suggestion
> that the process may have bumped into the open file limit. Whatever the
> problem was, it also prevented log collection, so we cannot confirm our
> guesses.
> Without knowing the underlying cause, we can still consider some potential
> improvements:
> 1. We probably should be catching non-IO exceptions in the append process. If
> the append to one of the indexes fails, we potentially truncate the log or
> re-throw it as an IOException to ensure that the log directory is no longer
> used.
> 2. Even without the unexpected exception, there is a small window during
> which even an IOException could lead to duplicate offsets. Marking a log
> directory offline is an asynchronous operation and there is no guarantee that
> another append cannot happen first. Given this, we probably need to detect
> and truncate duplicates during the log recovery process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
