Thanks for your reply. Yes, my original idea is that user can continue to collect the static information for reassignment as they are doing now. It is the status quo. I agree it can be beneficial to have a tool in Kafka to collect other information that may be needed for reassignment so that user would not need external tools at all. We can discuss this later when someone wants to lead the design and discussion of such a KIP.
I have some questions inline. On Tue, Aug 8, 2017 at 3:31 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > > > > Also, how do you think things would work in the context of KIP-179? > Would > > > the tool still invoke these requests or would it be done by the broker > > > receiving the alterTopics/reassignPartitions protocol call? > > > > > > > My gut feel is that the tool will still invoke these requests. But I > have a > > few questions to KIP-179 before I can answer this question. For example, > is > > AlterTopicsRequest request sent to controller only? If the new assignment > > is not written in zookeeper, how is this information propagated to the > new > > controller if the previous controller dies after it receives > > AlterTopicsRequest but before it sends LeaderAndIsrRequest? I can post > > these questions in that discussion thread later. > > > > > Let me answer here (though it's relevant to both KIPs): > > As I originally envisaged it, KIP-179's support for reassigning partitions > would have more-or-less taken the logic currently in the > ReassignPartitionsCommand (that is, writing JSON to the > ZkUtils.ReassignPartitionsPath) > and put it behind a suitable network protocol API. Thus it wouldn't matter > which broker received the protocol call: It would be acted on by brokers > being notified of the change in the ZK path, just as currently. This would > have kept the ReassignPartitionsCommand relatively simple, as it currently > is. > I am not sure I fully understand your proposal. I think you are saying that any broker can receive and handle the AlterTopicRequest. Let's say a non-controller broker received AlterTopicRequest, is this broker going to send LeaderAndIsrRequest to other brokers? Or is this broker create the reassignment znode in zookeper? I may have missed it. But I couldn't find the explanation of AlterTopicRequest handling in KIP-179. > > KIP-113 is obviously seeking to make more radical changes. The algorithm > described for moving a replica to a particular directory on a different > broker ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 113%3A+Support+replicas+movement+between+log+directories#KIP-113: > Supportreplicasmovementbetweenlogdirectories-2) > Howtoreassignreplicabetweenlogdirectoriesacrossbrokers > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 113%3A+Support+replicas+movement+between+log+directories#KIP-113: > Supportreplicasmovementbetweenlogdirectories-2% > 29Howtoreassignreplicabetweenlogdirectoriesacrossbrokers>) > involves both sending AlterReplicaDirRequest to "the" broker (the receiving > broker, I assume, but it's not spelled out), _as well as_ writing to the ZK > node. > > This assumes the script (ReassignPartitionsCommand) has direct access to > ZooKeeper, which is what KIP-179 is seeking to deprecate. It seems a waste > of time to put the logic in the script as part of KIP-113, only for KIP-179 > to have to move it back to the controller. > I am not sure I understand what you mean by "It seems a waste of time to put the logic in the script as part of KIP-113, only for KIP-179 to have to move it back to the controller". I assume that the logic you mentioned is "movement of replica to the specified log directory". This logic (or the implementation of this logic) resides mainly in the KafkaAdminClient and broker. The script only needs to parse the json file as appropriate and call the new API in AdminClient as appropriate. The logic in the script is therefore not much and can be easily moved to other classes if needed. Can you clarify why this logic, i.e. movement of replica to the specified log directory, needs to be moved to controller in KIP-179? I think it can still be done in the script and controller should not need to worry about log directory of any replica. Thanks, Dong
