GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/2986
KAFKA-5099: Replica Deletion Regression from KIP-101
Replica deletion regressed from KIP-101. Replica deletion happens when a
broker receives a StopReplicaRequest with delete=true. Ever since KAFKA-1911,
replica deletion has been async, meaning the broker responds with a
StopReplicaResponse simply after marking the replica directory as staged for
deletion. This marking happens by moving a data log directory and its contents
such as /tmp/kafka-logs1/t1-0 to a marked directory like
/tmp/kafka-logs1/t1-0.8c9c4c0c61c44cc59ebeb00075a2a07f-delete, acting as a
soft-delete. A scheduled thread later actually deletes the data. It appears
that the regression occurs while the scheduled thread is actually trying to
delete the data, which means the controller considers operations such as
partition reassignment and topic deletion complete. But if you look at the
log4j logs and data logs, you'll find that the soft-deleted data logs actually
won't get deleted.
The bug is that upon log deletion, we attempt to flush the
LeaderEpochFileCache to the original file location instead of the moved file
location. Restarting the broker actually allows for the soft-deleted
directories to get deleted.
This patch avoids the issue by simply not flushing the LeaderEpochFileCache
upon log deletion.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-5099
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2986.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2986
----
commit 081e09be262dea261f7faf9b5d81b50995600f68
Author: Onur Karaman <[email protected]>
Date: 2017-05-06T00:59:36Z
KAFKA-5099: Replica Deletion Regression from KIP-101
Replica deletion regressed from KIP-101. Replica deletion happens when a
broker receives a StopReplicaRequest with delete=true. Ever since KAFKA-1911,
replica deletion has been async, meaning the broker responds with a
StopReplicaResponse simply after marking the replica directory as staged for
deletion. This marking happens by moving a data log directory and its contents
such as /tmp/kafka-logs1/t1-0 to a marked directory like
/tmp/kafka-logs1/t1-0.8c9c4c0c61c44cc59ebeb00075a2a07f-delete, acting as a
soft-delete. A scheduled thread later actually deletes the data. It appears
that the regression occurs while the scheduled thread is actually trying to
delete the data, which means the controller considers operations such as
partition reassignment and topic deletion complete. But if you look at the
log4j logs and data logs, you'll find that the soft-deleted data logs actually
won't get deleted.
The bug is that upon log deletion, we attempt to flush the
LeaderEpochFileCache to the original file location instead of the moved file
location. Restarting the broker actually allows for the soft-deleted
directories to get deleted.
This patch avoids the issue by simply not flushing the LeaderEpochFileCache
upon log deletion.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
