Hi Jun, Thanks for taking a look. I can answer some questions here because I collaborated on this a bit, and David is on vacation for a few days.
On Wed, Oct 12, 2022, at 14:41, Jun Rao wrote: > Hi, David, > > Thanks for the KIP. A few comments below. > > 10. It's still not very clear to me how the KRaft controller works in the > dual writes mode to KRaft log and ZK when the brokers still run in ZK mode. > Does the KRaft controller run a ZK based controller in parallel or do we > derive what needs to be written to ZK based on KRaft controller logic? We derive what needs to be written to ZK based on KRaft controller logic. > I am also not sure how the KRaft controller handles broker > registration/deregistration, since brokers are still running in ZK mode and > are not heartbeating to the KRaft controller. The new controller will listen for broker registrations under /brokers. This is the only znode watch that the new controller will do. We did consider changing how ZK-based broker registration worked, but it just ended up being too much work for not enough gain. > > 12. "A new set of nodes will be provisioned to host the controller quorum." > I guess we don't support starting the KRaft controller quorum on existing > brokers. It would be useful to make that clear. > Agreed > 13. "Once the quorum is established and a leader is elected, the controller > will check the state of the cluster using the MigrationCheck RPC." How does > the quorum controller detect other brokers? Does the controller node need > to be configured with ZK connection string? If so, it would be useful to > document the additional configs that the quorum controller needs to set. > Yes, the controllers monitor ZK for broker registrations, as I mentioned above. So they need zk.connect and the other ZK connection configurations. > 14. "In order to prevent further writes to ZK, the first thing the new > KRaft quorum must do is take over leadership of the ZK controller. " The ZK > controller processing changes to /controller update asynchronously. How > does the KRaft controller know when the ZK controller has resigned before > it can safely copy the ZK data? > This should be done through expectedControllerEpochZkVersion, just like in ZK mode, right? We should bump this epoch value so that any writes from the old controller will not go through. I agree we should spell this out in the KIP. > 15. We have the following sentences. One says ControllerId is a random > KRaft broker and the other says it's the active controller. Which one is > correct? > "UpdateMetadata: for certain metadata changes, the KRaft controller will > need to send UpdateMetadataRequests to the ZK brokers. For the > “ControllerId” field in this request, the controller should specify a > random KRaft broker." > "In the UpdateMetadataRequest sent by the KRaft controller to the ZK > brokers, the ControllerId will point to the active controller which will be > used for the inter-broker requests." > Yeah, this seems like an error to me as well. A random value is not really useful. Plus the text here is self-contradictory, as you pointed out. I suspect what we should do here is add a new field, KRaftControllerId, and populate it with the real controller ID, and leave the old controllerId field as -1. A ZK-based broker that sees this can then consult its controller.quorum.voters configuration to see where it should send controller-bound RPCs. That (static) configuration lets us map between controller ID and host:port. We should still keep our existing epoch logic for deciding when UpdateMetadataRequest / LeaderAndIsrRequests are stale, with the caveat that any kraft-based epoch should be treated as greater than any ZK-based epoch. After all, the kraft epoch is coming from the epoch of __cluster_metadata, whereas the ZK epoch comes from ZK. > > 16. "Additionally, the controller must specify if a broker in “LiveBrokers” > is KRaft or ZK." Does that require any protocol changes to UpdateMetadata? > Yeah, I am also curious why the we need to care whether brokers are ZK or KRaft in UpdateMetadataRequest. We don't reveal this to clients, so can we just leave this out? best, Colin > Thanks, > > Jun > > On Wed, Oct 5, 2022 at 10:07 AM Mickael Maison <mickael.mai...@gmail.com> > wrote: > >> Hi David, >> >> Thanks for starting this important KIP. >> >> I've just taken a quick look so far but I've got a couple of initial >> questions: >> >> 1) What happens if a non KRaft compatible broker (or with >> kafka.metadata.migration.enable set to false) joins the cluster after >> the migration is triggered? >> >> 2) In the Failure Modes section you mention a scenario where a write >> to ZK fails. What happens when the divergence limit is reached? Is >> this a fatal condition? How much divergence should we allow? >> >> Thanks, >> Mickael >> >> On Wed, Oct 5, 2022 at 12:20 AM David Arthur <mum...@gmail.com> wrote: >> > >> > Hey folks, I wanted to get the ball rolling on the discussion for the >> > ZooKeeper migration KIP. This KIP details how we plan to do an online >> > migration of metadata from ZooKeeper to KRaft as well as a rolling >> > upgrade of brokers to KRaft mode. >> > >> > The general idea is to keep KRaft and ZooKeeper in sync during the >> > migration, so both types of brokers can exist simultaneously. Then, >> > once everything is migrated and updated, we can turn off ZooKeeper >> > writes. >> > >> > This is a pretty complex KIP, so please take a look :) >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration >> > >> > Thanks! >> > David >>
