Hi All
We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to open
up to the community for approval.
Previously the admin passed a list of replicas to be throttled:
quota.replication.throttled.replicas =
[partId]:[replica],[partId]:[replica],[partId]:[replica] etc
The change is to split this into two properties. One that corresponds to the
leader-side throttle, and the other that corresponds to the follower-side
throttle.
quota.leader.replication.throttled.replicas =
[partId]:[replica],[partId]:[replica],[partId]:[replica]
quota.follower.replication.throttled.replicas =
[partId]:[replica],[partId]:[replica],[partId]:[replica]
This provides more control over the throttling process. It also helps us with a
common use case which I’ve described below, for those interested.
Please reply if you have any comments / issues / suggestions.
Thanks as ever.
Ben
Sample Use Case:
Say we are moving partition 0. It has replicas [104,107,109] moving to
[105,107,113]
So the leader could be any of [104,107,109] and we know we will be creating new
replicas on 105 & 113.
In the current mechanism, all we can do is apply both leader and follower
throttles to all 5: [104,107,109,105,113] which will mean the regular
replication traffic between (say 107 is leader) 107->104 and 107->109 will be
throttled also. This is potentially problematic.
What we really want to do is apply:
LeaderThrottle [104,107,109]
FollowerThrottle [105,113]
This way, during a rebalance, we that standard replication traffic will not be
throttled, but the rebalance will perform correctly if leaders move. One
subtlety is that, should the leader move to the “move destination” node, it
would no longer be throttled. But this is actually to our benefit in the normal
case.
