[
https://issues.apache.org/jira/browse/KUDU-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabriella Lotz updated KUDU-3828:
---------------------------------
Description:
When the replica being moved is the tablet's leader,
{{AutoRebalancerTask::CheckMoveCompleted()}}
({{{}auto_rebalancer.cc:1067-1092{}}}) asks it to step down so it can be
evicted. Unlike the CLI rebalancer ({{{}tool_replica_util.cc:305-313{}}}), it
has two problems:
- It steps down right away, before the new replica has been promoted to voter.
The comment above the code says this only slows the move down. In practice it
adds an extra leader election, which clients feel.
- It treats any error from the step-down as a failed move. The next check
often sends the step-down again while the first one is still in progress, and
the tablet server rejects it with "leadership transfer ... already in
progress". The move is then dropped and its REPLACE marker cleared, even though
it was going fine. The new replica still gets promoted, so the tablet briefly
has 4 voters, and the master may evict a different replica than the one the
rebalancer meant to move.
There's no data loss, and the tablet gets back to its replication factor on its
own. But in my testing every move of a leader replica failed this way, which
shows up as failed moves and extra leader churn.
This has been in the code since the auto-rebalancer was first added.
Proposed fix: match the CLI. Only step down once the destination is a voter (or
there's no destination), and treat the step-down as best effort: log the error
and retry on the next check instead of failing the move.
was:
When the replica being moved is the tablet's leader,
\{{AutoRebalancerTask::CheckMoveCompleted()}}
(\{{auto_rebalancer.cc:1067-1092}}) asks it to step down so it can be evicted.
Unlike the CLI rebalancer (\{{tool_replica_util.cc:305-313}}), it has two
problems:
- It steps down right away, before the new replica has been promoted to voter.
The comment above the code says this only slows the move down. In practice it
adds an extra leader election, which clients feel.
- It treats any error from the step-down as a failed move. The next check often
sends the step-down again while the first one is still in progress, and the
tablet server rejects it with "leadership transfer ... already in progress".
The move is then dropped and its REPLACE marker cleared, even though it was
going fine. The new replica still gets promoted, so the tablet briefly has 4
voters, and the master may evict a different replica than the one the
rebalancer meant to move.
There's no data loss, and the tablet gets back to its replication factor on its
own. But in my testing every move of a leader replica failed this way, which
shows up as failed moves and extra leader churn.
This has been in the code since the auto-rebalancer was first added.
Proposed fix: match the CLI. Only step down once the destination is a voter (or
there's no destination), and treat the step-down as best effort: log the error
and retry on the next check instead of failing the move.
Reproduced by \{{AutoRebalancerTest.LeaderReplicaMoveCompletes}}, which moves
the leader's replica. It fails on master every time with the "already in
progress" error.
> Auto-rebalancer fails moves of a leader replica on step-down errors
> -------------------------------------------------------------------
>
> Key: KUDU-3828
> URL: https://issues.apache.org/jira/browse/KUDU-3828
> Project: Kudu
> Issue Type: Bug
> Reporter: Gabriella Lotz
> Priority: Major
>
> When the replica being moved is the tablet's leader,
> {{AutoRebalancerTask::CheckMoveCompleted()}}
> ({{{}auto_rebalancer.cc:1067-1092{}}}) asks it to step down so it can be
> evicted. Unlike the CLI rebalancer ({{{}tool_replica_util.cc:305-313{}}}), it
> has two problems:
> - It steps down right away, before the new replica has been promoted to
> voter. The comment above the code says this only slows the move down. In
> practice it adds an extra leader election, which clients feel.
> - It treats any error from the step-down as a failed move. The next check
> often sends the step-down again while the first one is still in progress, and
> the tablet server rejects it with "leadership transfer ... already in
> progress". The move is then dropped and its REPLACE marker cleared, even
> though it was going fine. The new replica still gets promoted, so the tablet
> briefly has 4 voters, and the master may evict a different replica than the
> one the rebalancer meant to move.
> There's no data loss, and the tablet gets back to its replication factor on
> its own. But in my testing every move of a leader replica failed this way,
> which shows up as failed moves and extra leader churn.
> This has been in the code since the auto-rebalancer was first added.
> Proposed fix: match the CLI. Only step down once the destination is a voter
> (or there's no destination), and treat the step-down as best effort: log the
> error and retry on the next check instead of failing the move.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)