Gabriella Lotz created KUDU-3828:
------------------------------------

             Summary: Auto-rebalancer fails moves of a leader replica on 
step-down errors
                 Key: KUDU-3828
                 URL: https://issues.apache.org/jira/browse/KUDU-3828
             Project: Kudu
          Issue Type: Bug
            Reporter: Gabriella Lotz


When the replica being moved is the tablet's leader, 
\{{AutoRebalancerTask::CheckMoveCompleted()}} 
(\{{auto_rebalancer.cc:1067-1092}}) asks it to step down so it can be evicted. 
Unlike the CLI rebalancer (\{{tool_replica_util.cc:305-313}}), it has two 
problems:

- It steps down right away, before the new replica has been promoted to voter. 
The comment above the code says this only slows the move down. In practice it 
adds an extra leader election, which clients feel.
- It treats any error from the step-down as a failed move. The next check often 
sends the step-down again while the first one is still in progress, and the 
tablet server rejects it with "leadership transfer ... already in progress". 
The move is then dropped and its REPLACE marker cleared, even though it was 
going fine. The new replica still gets promoted, so the tablet briefly has 4 
voters, and the master may evict a different replica than the one the 
rebalancer meant to move.

There's no data loss, and the tablet gets back to its replication factor on its 
own. But in my testing every move of a leader replica failed this way, which 
shows up as failed moves and extra leader churn.

This has been in the code since the auto-rebalancer was first added.

Proposed fix: match the CLI. Only step down once the destination is a voter (or 
there's no destination), and treat the step-down as best effort: log the error 
and retry on the next check instead of failing the move.

Reproduced by \{{AutoRebalancerTest.LeaderReplicaMoveCompletes}}, which moves 
the leader's replica. It fails on master every time with the "already in 
progress" error.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to