[
https://issues.apache.org/jira/browse/KAFKA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844064#comment-16844064
]
ASF GitHub Bot commented on KAFKA-8395:
---------------------------------------
axiak commented on pull request #6772: KAFKA-8395: Add the ability to back up
segment files on truncation
URL: https://github.com/apache/kafka/pull/6772
This adds the ability to rename/back up segment files on truncation
to zero. This is useful in the rare case that offset conflict resolution
results in an offset reset, which can result in data loss.
To enable, turn set `segment.backup.on.truncate.to.zero` to true
in the configuration.
We have unit tests included here and are starting
to stress test a working cluster to see if we can
reproduce the issue in a live environment.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add an ability to backup log segment files on truncation
> --------------------------------------------------------
>
> Key: KAFKA-8395
> URL: https://issues.apache.org/jira/browse/KAFKA-8395
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 2.2.0
> Reporter: Michael Axiak
> Priority: Minor
> Fix For: 2.2.1
>
>
> At HubSpot, we believe we hit a combination of bugs [1] [2], which may have
> caused us to lose data. In this scenario, as part of metadata conflict
> resolution a slowly starting up broker recovered an offset of zero and
> truncated segment files.
> As part of a belt-and-suspenders approach to reducing this risk in the
> future, I propose adding the ability to rename/backup these files and
> allowing kafka to move on. Note that this breaks the ordering guarantees, but
> allows one to recover the data and decide later how to approach it.
> This feature should be turned off by default but enabled with a configuration
> option.
> (A pull request is following soon on Github)
> 1: https://issues.apache.org/jira/browse/KAFKA-2178
> 2: https://issues.apache.org/jira/browse/KAFKA-1120
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
