Hi Luke, Thanks for the feedback. See my comments below.
I have also updated the KIP to reflect this discussion. On Wed, Feb 28, 2024 at 4:34 AM Luke Chen <show...@gmail.com> wrote: > > > 2. After "RemoveVoter", what is the role of the node? > > It looks like after the voter got removed from the voter set, it is not a > > voter anymore. But I think it can still fetch with the leader. So it > should > > be an observer, with a "process.role=controller"? And if the node was > > originally "process.role=controller,broker", it'll become a broker-only > > node? > > > Kafka nodes need to allow for controllers that are not voters. I don't > expect too many issues from an implementation point of view. Most of > it may just be aggressive validation in KafkaConfig. I think the > easier way to explain this state is that there will be controllers > that will never become active controllers. If we want, we can have a > monitor that turns on (1) if a node is in this state. What do you > think? > > I agree we have a way for users to monitor the node state, like when does > the controller completed the voter removal ( so that it is safe to be > shutdown), or when does the controller completed the voter addition (so > that users can start to add another controller), etc. We have the following metrics in the monitoring section: name=IsObserver;type=KafkaController - 1 if a controller node is not a voter for the KRaft cluster metadata partition, 0 otherwise. name=pending-voter-change;type=raft-metrics - 1 if there is a pending voter change that has not been committed, 0 otherwise. All of the Kafka nodes will report this metric. In addition to the metrics, the user can also use "kafka-metadata-quorum describe --status" command to query for the current voters set and the last committed voters set. The KRaft leader will check that it is safe to add or remove voters. The KRaft only allows for one uncommitted VotersRecord and an uncommitted VotersRecord can only change the previously committed VotersRecord by at most one change (either one addition or one removal). > 10. controller.quorum.voters: > This is an existing configuration. This configuration describes the state > of the quorum and will only be used if the kraft.version feature is 0. > > From the discussion, it looks like even if the kraft.version is 1, we > still first check the `controller.quorum.voters` if > `controller.quorum.bootstrap.servers` is not set. Is that correct? If so, > maybe we need to update the description? That's correct. In the current KRaft implementation controller.quorum.voters plays two roles: 1) describes the static voters set 2) the endpoint(s) that can be used to discover the leader. This KIP replaces 1) with the VotersRecord written to the log and snapshot and replaces 2) with the controller.quorum.bootstrap.servers configuration property. I updated the description for these configurations to make this clearer. > 11. When a controller starts up before joining as a voter, it'll be an > observer. In this case, will it be shown in the observer field of > `kafka-metadata-quorum describe --status`? Same question to a controller > after getting removed. Correct. The name=IsObserver;type=KafkaController metric for that node will also be 1. Note that the KRaft leader's reporting of observers is best effort. As long as the observer is fetching, the KRaft leader will report their state. If an observer doesn't Fetch for some time (I forget the value we use) the KRaft leader will remove the observer's replication state from its in-memory bookkeeping. > 12. What will happen if there's only 1 voter and user still tries to remove > the voter? Any error returned? Good point. This should probably be an error. If they are interested in decommissioning a cluster, they should instead just destroy the controller node. For this case, I updated the KIP to return the INVALID_REQUEST error. Thanks, -- -José
