Hi Viktor,
There is already KIP-236 for the same feature and George made a PR
for this as well.
Lets consolidate these two discussions. If you have any cases that are not
being solved by KIP-236 can you please mention them in that thread. We can
address as part of KIP-236.
Thanks,
Harsha
On Fri, Feb 22, 2019, at 5:44 AM, Viktor Somogyi-Vass wrote:
> Hi Folks,
>
> I've created a KIP about an improvement of the reassignment algorithm we
> have. It aims to enable partition-wise incremental reassignment. The
> motivation for this is to avoid excess load that the current replication
> algorithm implicitly carries as in that case there are points in the
> algorithm where both the new and old replica set could be online and
> replicating which puts double (or almost double) pressure on the brokers
> which could cause problems.
> Instead my proposal would slice this up into several steps where each step
> is calculated based on the final target replicas and the current replica
> assignment taking into account scenarios where brokers could be offline and
> when there are not enough replicas to fulfil the min.insync.replica
> requirement.
>
> The link to the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment
>
> I'd be happy to receive any feedback.
>
> An important note is that this KIP and another one, KIP-236 that is
> about
> interruptible reassignment (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment)
> should be compatible.
>
> Thanks,
> Viktor
>
