[
https://issues.apache.org/jira/browse/KAFKA-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-858:
--------------------------------
Attachment: kafka-858.patch
-This is a corner case bug that shows up only if, after the controlled shutdown
is issued to the broker and before the broker actually shuts down, the high
watermark checkpoint thread wakes up and writes the high watermarks.
- Since we depend on allPartitions map to checkpoint high watermarks, I think
the safest and least intrusive fix is to never remove partitions from that map.
If we do, then we run into a risk of overwriting high watermark values for a
partition.
- If we do the above, we have to take care that the partition is essentially
dormant from the point of view of the broker. So getPartition() should not
return partitions that have been stopped during controlled shutdown. This will
take care of failing future produce/fetching or having purgatory return the
proper error code. So I added a stoppedPartitions set that maintains the list
of stopped partitions.
- PartitionCount mbean will show only the alive partitions
- Stop replica request is half baked right now, but we don't need to focus on
fixing it since those changes will be pretty big, intrusive and risky. All we
need to make sure is that when stop replica request is received, we stop the
fetcher and remove the partition from leaderPartitions, so the broker does not
attempt to shrink isr for such partitions.
- Apart from the fix, I cleaned up logging in two places -
1. adding/removing fetchers to print partition the way the rest of the code
does. This is required for ease of grepping for a particular partition
2. state change log during leader/follower request to remove redundant entries
> High watermark values can be overwritten during controlled shutdown
> -------------------------------------------------------------------
>
> Key: KAFKA-858
> URL: https://issues.apache.org/jira/browse/KAFKA-858
> Project: Kafka
> Issue Type: Bug
> Components: replication
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: kafka-0.8, p1
> Attachments: kafka-858.patch
>
>
> Race condition between controlled shutdown, actual process shutdown and high
> watermark checkpoint thread frequency can cause high watermark values for a
> subset of partitions to be overwritten. So even if the controller sends a
> complete list of partitions to the broker, the highwatermark for some
> partitions is still 0. This causes the follower to fetch from the leader's
> start offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
