Hello, I've begun migrating some of my Zookeeper Kafka clusters to KRaft. A behaviour I've noticed twice across two different kafka cluster environments is after provisioning a kraft controller quorum in migration mode, it is possible for a kafka broker to become an active controller alongside a kraft controller broker.
For example, here are the steps I follow and the behaviour I notice (I'm currently using Kafka v3.6): 1. Enable the KRaft migration on the existing Kafka brokers (set the `controller.quorum.voter`, `controller.listener.names` and `zookeeper.metadata.migration.enable` configs in the server.properties file). 2. Deploy a kraft controller statefulset and service with the migration enabled so that data is copied over from Zookeeper and we enter a dual-write mode. 3. After a few minutes, I see that the migration has completed (it's a pretty small cluster). At this point, the kraft controller pod has been elected to be the controller (and I see this in zookeeper when I run `get /controller`). If the kafka brokers or kraft controller pods are restarted at any point after the migration is completed, a kafka broker is elected to be the controller and is reflected in zookeeper as well. Now, I have two active controllers - 1 is a kafka broker and 1 is a kraft controller broker. A couple questions I have: 1. Is this the expected behaviour? If so, how long after a migration has been completed should we hold off on restarting kafka brokers to avoid this situation? 2. Why is it possible for a kafka broker to be a controller again post-migration? 3. How do we come back to a state where a kraft controller broker is the only controller again in the least disruptive way possible? Thank you, Sanaa
