On Tue, Oct 5, 2021 at 8:53 AM David Arthur <davidart...@apache.org> wrote: > 2. Generate snapshot on downgrade > > > Metadata snapshot is generated and sent to the other inactive > > controllers and to brokers (this snapshot may be lossy!) > > Why do we need to send this downgraded snapshot to the brokers? The > > replicas have seen the FeatureLevelRecord and noticed the downgrade. > > Can we have the replicas each independently generate a downgraded > > snapshot at the offset for the downgraded FeatureLevelRecord? I assume > > that the active controller will guarantee that all records after the > > FatureLevelRecord use the downgraded version. If so, it would be good > > to mention that explicitly. > > > Similar to above, yes a broker that detects a downgrade via > FeatureLevelRecord could generate its own downgrade snapshot and reload its > state from that. This does get a little fuzzy when we consider cases where > brokers are on different software versions and could be generating a > downgrade snapshot for version X, but using different versions of the code. > It might be safer to let the controller generate the snapshot so each > broker (regardless of software version) gets the same records. However, for > upgrades (or downgrades) we expect the whole cluster to be running the same > software version before triggering the metadata.version change, so perhaps > this isn't a likely scenario. Thoughts? > >
Are you saying that for metadata.version X different software versions could generate different snapshots? If so, I would consider this an implementation bug, no? The format and content of a snapshot is a public API that needs to be supported across software versions. > 3. Max metadata version > > >For the first release that supports metadata.version, we can simply > > initialize metadata.version with the current (and only) version. For future > > releases, we will need a mechanism to bootstrap a particular version. This > > could be done using the meta.properties file or some similar mechanism. The > > reason we need the allow for a specific initial version is to support the > > use case of starting a Kafka cluster at version X with an older > > metadata.version. > > > I assume that the Active Controller will learn the metadata version of > > the broker through the BrokerRegistrationRequest. How will the Active > > Controller learn about the max metadata version of the inactive > > controller nodes? We currently don't send a registration request from > > the inactive controller to the active controller. > > > This came up during the design, but I neglected to add it to the KIP. We > will need a mechanism for determining the supported features of each > controller similar to how brokers use BrokerRegistrationRequest. Perhaps > controllers could write a FeatureLevelRecord (or similar) to the metadata > log indicating their supported version. WDYT? > Hmm. So I think you are proposing the following flow: 1. Cluster metadata partition replicas establish a quorum using ApiVersions and the KRaft protocol. 2. Inactive controllers send a registration RPC to the active controller. 3. The active controller persists this information to the metadata log. What happens if the inactive controllers send a metadata.version range that is not compatible with the metadata.version set for the cluster? > Why do you need to bootstrap a particular version? Isn't the intent > > that the broker will learn the active metadata version by reading the > > metadata before unfencing? > > > This bootstrapping is needed for when a KRaft cluster is first started. If > we don't have this mechanism, the cluster can't really do anything until > the operator finalizes the metadata.version with the tool. The > bootstrapping will be done by the controller and the brokers will see this > version as a record (like you say). I'll add some text to clarify this. > > Got it. A new cluster will use the metadata.version of the first active controller. The first active controller will persist this information on the metadata log. All replicas (inactive controller and brokers) will configure themselves based on this record. > 4. Reject Registration - This is related to the bullet point above. > > What will be the behavior of the active controller if the broker sends > > a metadata version that is not compatible with the cluster wide > > metadata version? > > > If a broker starts up with a lower supported version range than the current > cluster metadata.version, it should log an error and shutdown. This is in > line with KIP-584. > Okay. We need to extend this for the controller case. Thanks -- -Jose
