[
https://issues.apache.org/jira/browse/KAFKA-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749810#comment-16749810
]
Ismael Juma commented on KAFKA-7854:
------------------------------------
There is already a KIP for doing reassignment and cancellation via admin APIs.
That is definitely the way forward. If you look at the reassign tool, it's very
clear that the intent was for the znode not to be updated by the tool unless it
was empty.
> Behavior change in controller picking up partition reassignment tasks since
> 1.1.0
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-7854
> URL: https://issues.apache.org/jira/browse/KAFKA-7854
> Project: Kafka
> Issue Type: Improvement
> Components: controller
> Reporter: Zhanxiang (Patrick) Huang
> Priority: Major
>
> After [https://github.com/apache/kafka/pull/4143,] the controller does not
> subscribe to data change on /admin/reassign_partitions any more (in order to
> avoid unnecessarily loading the reassignment data again after controller
> updating the znode) as opposed to the previous kafka versions. However, there
> are systems built around kafka relying on the previous behavior to
> incrementally update the list of partition reassignment since kafka does not
> natively support that.
>
> For example, [cruise control|https://github.com/linkedin/cruise-control] can
> rely on the previous behavior (controller listening to data changes) to
> maintain the reassignment concurrency by dynamically updating the data in the
> reassignment znode instead of waiting for the current batch to finish and
> doing reassignment batch by batch, which can significantly reduce the
> rebalance time in production clusters. Although directly updating the znode
> can somehow be viewed as an anti-pattern in the long term, this is necessary
> since kafka does not natively support incrementally submit more reassignment
> tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise
> control no longer works because the controller behavior has changed. This
> reveals the following problems:
> * These behavior changes may be viewed as internal changes so compatibility
> is not guaranteed but I think by convention people do view this as public
> interfaces and rely on the compatibility. In this case, I think we should
> clearly document the data contract for the partition reassignment task to
> avoid misusage and making controller changes that break the defined data
> contract. There may be other cases (e.g. topic deletion) whose data contracts
> need to be clearly defined and we should keep it in mind when making
> controller changes.
> * Kafka does not natively support incrementally submit more reassignment
> tasks. If we do want to support that nicely, we should consider change how we
> store the reassignment data to store the data in child nodes and let the
> controller listen on child node changes, similar to what we do for
> /admin/delete_topics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
