Gabriella Lotz created KUDU-3828:
------------------------------------
Summary: Auto-rebalancer fails moves of a leader replica on
step-down errors
Key: KUDU-3828
URL: https://issues.apache.org/jira/browse/KUDU-3828
Project: Kudu
Issue Type: Bug
Reporter: Gabriella Lotz
When the replica being moved is the tablet's leader,
\{{AutoRebalancerTask::CheckMoveCompleted()}}
(\{{auto_rebalancer.cc:1067-1092}}) asks it to step down so it can be evicted.
Unlike the CLI rebalancer (\{{tool_replica_util.cc:305-313}}), it has two
problems:
- It steps down right away, before the new replica has been promoted to voter.
The comment above the code says this only slows the move down. In practice it
adds an extra leader election, which clients feel.
- It treats any error from the step-down as a failed move. The next check often
sends the step-down again while the first one is still in progress, and the
tablet server rejects it with "leadership transfer ... already in progress".
The move is then dropped and its REPLACE marker cleared, even though it was
going fine. The new replica still gets promoted, so the tablet briefly has 4
voters, and the master may evict a different replica than the one the
rebalancer meant to move.
There's no data loss, and the tablet gets back to its replication factor on its
own. But in my testing every move of a leader replica failed this way, which
shows up as failed moves and extra leader churn.
This has been in the code since the auto-rebalancer was first added.
Proposed fix: match the CLI. Only step down once the destination is a voter (or
there's no destination), and treat the step-down as best effort: log the error
and retry on the next check instead of failing the move.
Reproduced by \{{AutoRebalancerTest.LeaderReplicaMoveCompletes}}, which moves
the leader's replica. It fails on master every time with the "already in
progress" error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)