[
https://issues.apache.org/jira/browse/KAFKA-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730294#comment-13730294
]
Neha Narkhede edited comment on KAFKA-999 at 8/6/13 3:03 AM:
-------------------------------------------------------------
Sriram, Ack you are right. Forgot to mention that caveat in the bug description.
Swapnil,
The problem with undoing changes to controllerContext and removing the broker
if the controlled shutdown request failed, is that there is a risk of
mistakenly moving leaders to that broker. The reason is because even if the
controlled shutdown attempt failed, it goes ahead with an uncontrolled
shutdown. In that case, there is no value to letting it become a leader for
more partitions since the point of controlled shutdown is to move existing
leaders off.
Split brain is always a problem that is both dangerous and causes unnecessary
failures (e.g. consumer rebalancing). The purpose of a controller in Kafka is
to be the only brain in the cluster and make state change decisions based on a
single global view of the system. In this case, even if we keep the extra check
on the broker, the leader can fail immediately after the if statement succeeds.
So that check doesn't help anyways (Jun had pointed that out to me in the past,
but I wasn't convinced back then :) ). Also, even if the leader fails, the
broker should depend on the controller to do the right thing and send another
leader and isr request with a new leader. That way any changes to the
controller logic (new features like controlled shutdown) will not cause
unexpected side effects such as this bug.
was (Author: nehanarkhede):
Sriram, Ack you are right. Forgot to mention that caveat in the bug
description.
Swapnil,
The problem with undoing changes to controllerContext and removing the broker
if the controlled shutdown request failed, is that there is a risk of
mistakenly moving leaders to that broker. The reason is because even if the
controlled shutdown attempt failed, it goes ahead with an uncontrolled
shutdown. In that case, there is no value to letting it become leaders for more
partitions since the point of controlled shutdown is to move existing leaders
off.
Split brain is always a problem that is both dangerous and causes unnecessary
failures (e.g. consumer rebalancing). The purpose of a controller in Kafka is
to be the only brain in the cluster and make state change decisions based on a
single global view of the system. In this case, even if we keep the extra check
on the broker, the leader can fail immediately after the if statement succeeds.
So that check doesn't help anyways (Jun had pointed that out to me in the past,
but I wasn't convinced back then :) ). Also, even if the leader fails, the
broker should depend on the controller to do the right thing and send another
leader and isr request with a new leader. That way any changes to the
controller logic (new features like controlled shutdown) will not cause
unexpected side effects such as this bug.
> Controlled shutdown never succeeds until the broker is killed
> -------------------------------------------------------------
>
> Key: KAFKA-999
> URL: https://issues.apache.org/jira/browse/KAFKA-999
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Critical
>
> A race condition in the way leader and isr request is handled by the broker
> and controlled shutdown can lead to a situation where controlled shutdown can
> never succeed and the only way to bounce the broker is to kill it.
> The root cause is that broker uses a smart to avoid fetching from a leader
> that is not alive according to the controller. This leads to the broker
> aborting a become follower request. And in cases where replication factor is
> 2, the leader can never be transferred to a follower since it keeps rejecting
> the become follower request and stays out of the ISR. This causes controlled
> shutdown to fail forever
> One sequence of events that led to this bug is as follows -
> - Broker 2 is leader and controller
> - Broker 2 is bounced (uncontrolled shutdown)
> - Controller fails over
> - Controlled shutdown is invoked on broker 1
> - Controller starts leader election for partitions that broker 2 led
> - Controller sends become follower request with leader as broker 1 to broker
> 2. At the same time, it does not include broker 1 in alive broker list sent
> as part of leader and isr request
> - Broker 2 rejects leaderAndIsr request since leader is not in the list of
> alive brokers
> - Broker 1 fails to transfer leadership to broker 2 since broker 2 is not in
> ISR
> - Controlled shutdown can never succeed on broker 1
> Since controlled shutdown is a config option, if there are bugs in controlled
> shutdown, there is no option but to kill the broker
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
