Hey Jun, I added a section on "Cluster Bootstrapping" which discusses clusterId generation and the process through which brokers find the current leader. The quick summary is that the first controller will be responsible for generating the clusterId and persisting it in the metadata log. Before the first leader has been elected, quorum APIs will skip clusterId validation. This seems reasonable since this is primarily intended to prevent the damage from misconfiguration after a cluster has been running for some time. Upon startup, brokers begin by sending Fetch requests to find the current leader. This will include the cluster.id from meta.properties if it is present. The broker will shutdown immediately if it receives INVALID_CLUSTER_ID from the Fetch response.
I also added some details about our testing strategy, which you asked about previously. Thanks, Jason On Mon, Jul 27, 2020 at 10:46 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > On Mon, Jul 27, 2020 at 4:58 AM Unmesh Joshi <unmeshjo...@gmail.com> > wrote: > > > Just checked etcd and zookeeper code, and both support leader to step > down > > as a follower to make sure there are no two leaders if the leader has > been > > disconnected from the majority of the followers > > For etcd this is https://github.com/etcd-io/etcd/issues/3866 > > For Zookeeper its https://issues.apache.org/jira/browse/ZOOKEEPER-1699 > > I was just thinking if it would be difficult to implement in the Pull > based > > model, but I guess not. It is possibly the same way ISR list is managed > > currently, if leader of the controller quorum loses majority of the > > followers, it should step down and become follower, that way, telling > > client in time that it was disconnected from the quorum, and not keep on > > sending state metadata to clients. > > > > Thanks, > > Unmesh > > > > > > On Mon, Jul 27, 2020 at 9:31 AM Unmesh Joshi <unmeshjo...@gmail.com> > > wrote: > > > > > >>Could you clarify on this question? Which part of the raft group > > doesn't > > > >>know about leader dis-connection? > > > The leader of the controller quorum is partitioned from the controller > > > cluster, and a different leader is elected for the remaining controller > > > cluster. > > > I see your concern. For KIP-595 implementation, since there is no regular > heartbeats sent > from the leader to the followers, we decided to piggy-back on the fetch > timeout so that if the leader did not receive Fetch > requests from a majority of the quorum for that amount of time, it would > begin a new election and > start sending VoteRequest to voter nodes in the cluster to understand the > latest quorum. You could > find more details in this section > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-Vote > > > . > > > > > I think there are two things here, > > > 1. The old leader will not know if it's disconnected from the rest of > > the > > > controller quorum cluster unless it receives BeginQuorumEpoch from the > > new > > > leader. So it will keep on serving stale metadata to the clients > > (Brokers, > > > Producers and Consumers) > > > 2. I assume, the Broker Leases will be managed on the controller quorum > > > leader. This partitioned leader will keep on tracking broker leases it > > has, > > > while the new leader of the quorum will also start managing broker > > leases. > > > So while the quorum leader is partitioned, there will be two membership > > > views of the kafka brokers managed on two leaders. > > > Unless broker heartbeats are also replicated as part of the Raft log, > > > there is no way to solve this? > > > I know LogCabin implementation does replicate client heartbeats. I > > suspect > > > that the same issue is there in Zookeeper, which does not replicate > > client > > > Ping requests.. > > > > > > Thanks, > > > Unmesh > > > > > > > > > > > > On Mon, Jul 27, 2020 at 6:23 AM Boyang Chen < > reluctanthero...@gmail.com> > > > wrote: > > > > > >> Thanks for the questions Unmesh! > > >> > > >> On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi <unmeshjo...@gmail.com> > > >> wrote: > > >> > > >> > Hi, > > >> > > > >> > In the FetchRequest Handling, how to make sure we handle scenarios > > where > > >> > the leader might have been disconnected from the cluster, but > doesn't > > >> know > > >> > yet? > > >> > > > >> Could you clarify on this question? Which part of the raft group > doesn't > > >> know about leader > > >> dis-connection? > > >> > > >> > > >> > As discussed in the Raft Thesis section 6.4, the linearizable > > semantics > > >> of > > >> > read requests is implemented in LogCabin by sending heartbeat to > > >> followers > > >> > and waiting till the heartbeats are successful to make sure that the > > >> leader > > >> > is still the leader. > > >> > I think for the controller quorum to make sure none of the consumers > > get > > >> > stale data, it's important to have linearizable semantics? In the > pull > > >> > based model, the leader will need to wait for heartbeats from the > > >> followers > > >> > before returning each fetch request from the consumer then? Or do we > > >> need > > >> > to introduce some other request? > > >> > (Zookeeper does not have linearizable semantics for read requests, > but > > >> as > > >> > of now all the kafka interactions are through writes and watches). > > >> > > > >> > This is a very good question. For our v1 implementation we are not > > >> aiming > > >> to guarantee linearizable read, which > > >> would be considered as a follow-up effort. Note that today in Kafka > > there > > >> is no guarantee on the metadata freshness either, > > >> so no regression is introduced. > > >> > > >> > > >> > Thanks, > > >> > Unmesh > > >> > > > >> > On Fri, Jul 24, 2020 at 11:36 PM Jun Rao <j...@confluent.io> wrote: > > >> > > > >> > > Hi, Jason, > > >> > > > > >> > > Thanks for the reply. > > >> > > > > >> > > 101. Sounds good. Regarding clusterId, I am not sure storing it in > > the > > >> > > metadata log is enough. For example, the vote request includes > > >> clusterId. > > >> > > So, no one can vote until they know the clusterId. Also, it would > be > > >> > useful > > >> > > to support the case when a voter completely loses its disk and > needs > > >> to > > >> > > recover. > > >> > > > > >> > > 210. There is no longer a FindQuorum request. When a follower > > >> restarts, > > >> > how > > >> > > does it discover the leader? Is that based on DescribeQuorum? It > > >> would be > > >> > > useful to document this. > > >> > > > > >> > > Jun > > >> > > > > >> > > On Fri, Jul 17, 2020 at 2:15 PM Jason Gustafson < > ja...@confluent.io > > > > > >> > > wrote: > > >> > > > > >> > > > Hi Jun, > > >> > > > > > >> > > > Thanks for the questions. > > >> > > > > > >> > > > 101. I am treating some of the bootstrapping problems as out of > > the > > >> > scope > > >> > > > of this KIP. I am working on a separate proposal which addresses > > >> > > > bootstrapping security credentials specifically. Here is a rough > > >> sketch > > >> > > of > > >> > > > how I am seeing it: > > >> > > > > > >> > > > 1. Dynamic broker configurations including encrypted passwords > > will > > >> be > > >> > > > persisted in the metadata log and cached in the broker's > > >> > > `meta.properties` > > >> > > > file. > > >> > > > 2. We will provide a tool which allows users to directly > override > > >> the > > >> > > > values in `meta.properties` without requiring access to the > > quorum. > > >> > This > > >> > > > can be used to bootstrap the credentials of the voter set itself > > >> before > > >> > > the > > >> > > > cluster has been started. > > >> > > > 3. Some dynamic config changes will only be allowed when a > broker > > is > > >> > > > online. For example, changing a truststore password dynamically > > >> would > > >> > > > prevent that broker from being able to start if it were offline > > when > > >> > the > > >> > > > change was made. > > >> > > > 4. I am still thinking a little bit about SCRAM credentials, but > > >> most > > >> > > > likely they will be handled with an approach similar to > > >> > > `meta.properties`. > > >> > > > > > >> > > > 101.3 As for the question about `clusterId`, I think the way we > > >> would > > >> > do > > >> > > > this is to have the first elected leader generate a UUID and > write > > >> it > > >> > to > > >> > > > the metadata log. Let me add some detail to the proposal about > > this. > > >> > > > > > >> > > > A few additional answers below: > > >> > > > > > >> > > > 203. Yes, that is correct. > > >> > > > > > >> > > > 204. That is a good question. What happens in this case is that > > all > > >> > > voters > > >> > > > advance their epoch to the one designated by the candidate even > if > > >> they > > >> > > > reject its vote request. Assuming the candidate fails to be > > elected, > > >> > the > > >> > > > election will be retried until a leader emerges. > > >> > > > > > >> > > > 205. I had some discussion with Colin offline about this > problem. > > I > > >> > think > > >> > > > the answer should be "yes," but it probably needs a little more > > >> > thought. > > >> > > > Handling JBOD failures is tricky. For an observer, we can > > replicate > > >> the > > >> > > > metadata log from scratch safely in a new log dir. But if the > log > > >> dir > > >> > of > > >> > > a > > >> > > > voter fails, I do not think it is generally safe to start from > an > > >> empty > > >> > > > state. > > >> > > > > > >> > > > 206. Yes, that is discussed in KIP-631 I believe. > > >> > > > > > >> > > > 207. Good suggestion. I will work on this. > > >> > > > > > >> > > > > > >> > > > Thanks, > > >> > > > Jason > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Thu, Jul 16, 2020 at 3:44 PM Jun Rao <j...@confluent.io> > wrote: > > >> > > > > > >> > > > > Hi, Jason, > > >> > > > > > > >> > > > > Thanks for the updated KIP. Looks good overall. A few more > > >> comments > > >> > > > below. > > >> > > > > > > >> > > > > 101. I still don't see a section on bootstrapping related > > issues. > > >> It > > >> > > > would > > >> > > > > be useful to document if/how the following is supported. > > >> > > > > 101.1 Currently, we support auto broker id generation. Is this > > >> > > supported > > >> > > > > for bootstrap brokers? > > >> > > > > 101.2 As Colin mentioned, sometimes we may need to load the > > >> security > > >> > > > > credentials to be broker before it can be connected to. Could > > you > > >> > > > provide a > > >> > > > > bit more detail on how this will work? > > >> > > > > 101.3 Currently, we use ZK to generate clusterId on a new > > cluster. > > >> > With > > >> > > > > Raft, how does every broker generate the same clusterId in a > > >> > > distributed > > >> > > > > way? > > >> > > > > > > >> > > > > 200. It would be useful to document if the various special > > offsets > > >> > (log > > >> > > > > start offset, recovery point, HWM, etc) for the Raft log are > > >> stored > > >> > in > > >> > > > the > > >> > > > > same existing checkpoint files or not. > > >> > > > > 200.1 Since the Raft log flushes every append, does that allow > > us > > >> to > > >> > > > > recover from a recovery point within the active segment or do > we > > >> > still > > >> > > > need > > >> > > > > to scan the full segment including the recovery point? The > > former > > >> can > > >> > > be > > >> > > > > tricky since multiple records can fall into the same disk page > > >> and a > > >> > > > > subsequent flush may corrupt a page with previously flushed > > >> records. > > >> > > > > > > >> > > > > 201. Configurations. > > >> > > > > 201.1 How do the Raft brokers get security related configs for > > >> inter > > >> > > > broker > > >> > > > > communication? Is that based on the existing > > >> > > > > inter.broker.security.protocol? > > >> > > > > 201.2 We have quorum.retry.backoff.max.ms and > > >> > quorum.retry.backoff.ms, > > >> > > > but > > >> > > > > only quorum.election.backoff.max.ms. This seems a bit > > >> inconsistent. > > >> > > > > > > >> > > > > 202. Metrics: > > >> > > > > 202.1 TotalTimeMs, InboundQueueTimeMs, HandleTimeMs, > > >> > > OutboundQueueTimeMs: > > >> > > > > Are those the same as existing totalTime, requestQueueTime, > > >> > localTime, > > >> > > > > responseQueueTime? Could we reuse the existing ones with the > tag > > >> > > > > request=[request-type]? > > >> > > > > 202.2. Could you explain what InboundChannelSize and > > >> > > OutboundChannelSize > > >> > > > > are? > > >> > > > > 202.3 ElectionLatencyMax/Avg: It seems that both should be > > >> windowed? > > >> > > > > > > >> > > > > 203. Quorum State: I assume that LeaderId will be kept > > >> consistently > > >> > > with > > >> > > > > LeaderEpoch. For example, if a follower transitions to > candidate > > >> and > > >> > > > bumps > > >> > > > > up LeaderEpoch, it will set leaderId to -1 and persist both in > > the > > >> > > Quorum > > >> > > > > state file. Is that correct? > > >> > > > > > > >> > > > > 204. I was thinking about a corner case when a Raft broker is > > >> > > partitioned > > >> > > > > off. This broker will then be in a continuous loop of bumping > up > > >> the > > >> > > > leader > > >> > > > > epoch, but failing to get enough votes. When the partitioning > is > > >> > > removed, > > >> > > > > this broker's high leader epoch will force a leader election. > I > > >> > assume > > >> > > > > other Raft brokers can immediately advance their leader epoch > > >> passing > > >> > > the > > >> > > > > already bumped epoch such that leader election won't be > delayed. > > >> Is > > >> > > that > > >> > > > > right? > > >> > > > > > > >> > > > > 205. In a JBOD setting, could we use the existing tool to move > > the > > >> > Raft > > >> > > > log > > >> > > > > from one disk to another? > > >> > > > > > > >> > > > > 206. The KIP doesn't mention the local metadata store derived > > from > > >> > the > > >> > > > Raft > > >> > > > > log. Will that be covered in a separate KIP? > > >> > > > > > > >> > > > > 207. Since this is a critical component. Could we add a > section > > on > > >> > the > > >> > > > > testing plan for correctness? > > >> > > > > > > >> > > > > 208. Performance. Do we plan to do group commit (e.g. buffer > > >> pending > > >> > > > > appends during a flush and then flush all accumulated pending > > >> records > > >> > > > > together in the next flush) for better throughput? > > >> > > > > > > >> > > > > 209. "the leader can actually defer fsync until it knows > > >> > "quorum.size - > > >> > > > 1" > > >> > > > > has get to a certain entry offset." Why is that "quorum.size - > > 1" > > >> > > instead > > >> > > > > of the majority of the quorum? > > >> > > > > > > >> > > > > Thanks, > > >> > > > > > > >> > > > > Jun > > >> > > > > > > >> > > > > On Mon, Jul 13, 2020 at 9:43 AM Jason Gustafson < > > >> ja...@confluent.io> > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Hi All, > > >> > > > > > > > >> > > > > > Just a quick update on the proposal. We have decided to move > > >> quorum > > >> > > > > > reassignment to a separate KIP: > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-642%3A+Dynamic+quorum+reassignment > > >> > > > > > . > > >> > > > > > The way this ties into cluster bootstrapping is complicated, > > so > > >> we > > >> > > felt > > >> > > > > we > > >> > > > > > needed a bit more time for validation. That leaves the core > of > > >> this > > >> > > > > > proposal as quorum-based replication. If there are no > further > > >> > > comments, > > >> > > > > we > > >> > > > > > will plan to start a vote later this week. > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Jason > > >> > > > > > > > >> > > > > > On Wed, Jun 24, 2020 at 10:43 AM Guozhang Wang < > > >> wangg...@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > > @Jun Rao <jun...@gmail.com> > > >> > > > > > > > > >> > > > > > > Regarding your comment about log compaction. After some > > >> > deep-diving > > >> > > > > into > > >> > > > > > > this we've decided to propose a new snapshot-based log > > >> cleaning > > >> > > > > mechanism > > >> > > > > > > which would be used to replace the current compaction > > >> mechanism > > >> > for > > >> > > > > this > > >> > > > > > > meta log. A new KIP will be proposed specifically for this > > >> idea. > > >> > > > > > > > > >> > > > > > > All, > > >> > > > > > > > > >> > > > > > > I've updated the KIP wiki a bit updating one config " > > >> > > > > > > election.jitter.max.ms" > > >> > > > > > > to "election.backoff.max.ms" to make it more clear about > > the > > >> > > usage: > > >> > > > > the > > >> > > > > > > configured value will be the upper bound of the binary > > >> > exponential > > >> > > > > > backoff > > >> > > > > > > time after a failed election, before starting a new one. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Guozhang > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Fri, Jun 12, 2020 at 9:26 AM Boyang Chen < > > >> > > > > reluctanthero...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Thanks for the suggestions Guozhang. > > >> > > > > > > > > > >> > > > > > > > On Thu, Jun 11, 2020 at 2:51 PM Guozhang Wang < > > >> > > wangg...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Hello Boyang, > > >> > > > > > > > > > > >> > > > > > > > > Thanks for the updated information. A few questions > > here: > > >> > > > > > > > > > > >> > > > > > > > > 1) Should the quorum-file also update to support > > >> multi-raft? > > >> > > > > > > > > > > >> > > > > > > > > I'm neutral about this, as we don't know yet how the > > >> > multi-raft > > >> > > > > > modules > > >> > > > > > > > would behave. If > > >> > > > > > > > we have different threads operating different raft > groups, > > >> > > > > > consolidating > > >> > > > > > > > the `checkpoint` files seems > > >> > > > > > > > not reasonable. We could always add `multi-quorum-file` > > >> later > > >> > if > > >> > > > > > > possible. > > >> > > > > > > > > > >> > > > > > > > 2) In the previous proposal, there's fields in the > > >> > > > FetchQuorumRecords > > >> > > > > > > like > > >> > > > > > > > > latestDirtyOffset, is that dropped intentionally? > > >> > > > > > > > > > > >> > > > > > > > > I dropped the latestDirtyOffset since it is associated > > >> with > > >> > the > > >> > > > log > > >> > > > > > > > compaction discussion. This is beyond this KIP scope and > > we > > >> > could > > >> > > > > > > > potentially get a separate KIP to talk about it. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > 3) I think we also need to elaborate a bit more > details > > >> > > regarding > > >> > > > > > when > > >> > > > > > > to > > >> > > > > > > > > send metadata request and discover-brokers; currently > we > > >> only > > >> > > > > > discussed > > >> > > > > > > > > during bootstrap how these requests would be sent. I > > think > > >> > the > > >> > > > > > > following > > >> > > > > > > > > scenarios would also need these requests > > >> > > > > > > > > > > >> > > > > > > > > 3.a) As long as a broker does not know the current > > quorum > > >> > > > > (including > > >> > > > > > > the > > >> > > > > > > > > leader and the voters), it should continue > periodically > > >> ask > > >> > > other > > >> > > > > > > brokers > > >> > > > > > > > > via "metadata. > > >> > > > > > > > > 3.b) As long as a broker does not know all the current > > >> quorum > > >> > > > > voter's > > >> > > > > > > > > connections, it should continue periodically ask other > > >> > brokers > > >> > > > via > > >> > > > > > > > > "discover-brokers". > > >> > > > > > > > > 3.c) When the leader's fetch timeout elapsed, it > should > > >> send > > >> > > > > metadata > > >> > > > > > > > > request. > > >> > > > > > > > > > > >> > > > > > > > > Make sense, will add to the KIP. > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > Guozhang > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > On Wed, Jun 10, 2020 at 5:20 PM Boyang Chen < > > >> > > > > > > reluctanthero...@gmail.com> > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Hey all, > > >> > > > > > > > > > > > >> > > > > > > > > > follow-up on the previous email, we made some more > > >> updates: > > >> > > > > > > > > > > > >> > > > > > > > > > 1. The Alter/DescribeQuorum RPCs are also > > re-structured > > >> to > > >> > > use > > >> > > > > > > > > multi-raft. > > >> > > > > > > > > > > > >> > > > > > > > > > 2. We add observer status into the > > >> DescribeQuorumResponse > > >> > as > > >> > > we > > >> > > > > see > > >> > > > > > > it > > >> > > > > > > > > is a > > >> > > > > > > > > > low hanging fruit which is very useful for user > > >> debugging > > >> > and > > >> > > > > > > > > reassignment. > > >> > > > > > > > > > > > >> > > > > > > > > > 3. The FindQuorum RPC is replaced with > DiscoverBrokers > > >> RPC, > > >> > > > which > > >> > > > > > is > > >> > > > > > > > > purely > > >> > > > > > > > > > in charge of discovering broker connections in a > > gossip > > >> > > manner. > > >> > > > > The > > >> > > > > > > > > quorum > > >> > > > > > > > > > leader discovery is piggy-back on the Metadata RPC > for > > >> the > > >> > > > topic > > >> > > > > > > > > partition > > >> > > > > > > > > > leader, which in our case is the single metadata > > >> partition > > >> > > for > > >> > > > > the > > >> > > > > > > > > version > > >> > > > > > > > > > one. > > >> > > > > > > > > > > > >> > > > > > > > > > Let me know if you have any questions. > > >> > > > > > > > > > > > >> > > > > > > > > > Boyang > > >> > > > > > > > > > > > >> > > > > > > > > > On Tue, Jun 9, 2020 at 11:01 PM Boyang Chen < > > >> > > > > > > > reluctanthero...@gmail.com> > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Hey all, > > >> > > > > > > > > > > > > >> > > > > > > > > > > Thanks for the great discussions so far. I'm > posting > > >> some > > >> > > KIP > > >> > > > > > > updates > > >> > > > > > > > > > from > > >> > > > > > > > > > > our working group discussion: > > >> > > > > > > > > > > > > >> > > > > > > > > > > 1. We will be changing the core RPCs from > > single-raft > > >> API > > >> > > to > > >> > > > > > > > > multi-raft. > > >> > > > > > > > > > > This means all protocols will be "batch" in the > > first > > >> > > > version, > > >> > > > > > but > > >> > > > > > > > the > > >> > > > > > > > > > KIP > > >> > > > > > > > > > > itself only illustrates the design for a single > > >> metadata > > >> > > > topic > > >> > > > > > > > > partition. > > >> > > > > > > > > > > The reason is to "keep the door open" for future > > >> > extensions > > >> > > > of > > >> > > > > > this > > >> > > > > > > > > piece > > >> > > > > > > > > > > of module such as a sharded controller or general > > >> quorum > > >> > > > based > > >> > > > > > > topic > > >> > > > > > > > > > > replication, beyond the current Kafka replication > > >> > protocol. > > >> > > > > > > > > > > > > >> > > > > > > > > > > 2. We will piggy-back on the current Kafka Fetch > API > > >> > > instead > > >> > > > of > > >> > > > > > > > > inventing > > >> > > > > > > > > > > a new FetchQuorumRecords RPC. The motivation is > > about > > >> the > > >> > > > same > > >> > > > > as > > >> > > > > > > #1 > > >> > > > > > > > as > > >> > > > > > > > > > > well as making the integration work easier, > instead > > of > > >> > > > letting > > >> > > > > > two > > >> > > > > > > > > > similar > > >> > > > > > > > > > > RPCs diverge. > > >> > > > > > > > > > > > > >> > > > > > > > > > > 3. In the EndQuorumEpoch protocol, instead of only > > >> > sending > > >> > > > the > > >> > > > > > > > request > > >> > > > > > > > > to > > >> > > > > > > > > > > the most caught-up voter, we shall broadcast the > > >> > > information > > >> > > > to > > >> > > > > > all > > >> > > > > > > > > > voters, > > >> > > > > > > > > > > with a sorted voter list in descending order of > > their > > >> > > > > > corresponding > > >> > > > > > > > > > > replicated offset. In this way, the top voter will > > >> > become a > > >> > > > > > > candidate > > >> > > > > > > > > > > immediately, while the other voters shall wait for > > an > > >> > > > > exponential > > >> > > > > > > > > > back-off > > >> > > > > > > > > > > to trigger elections, which helps ensure the top > > voter > > >> > gets > > >> > > > > > > elected, > > >> > > > > > > > > and > > >> > > > > > > > > > > the election eventually happens when the top voter > > is > > >> not > > >> > > > > > > responsive. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Please see the updated KIP and post any questions > or > > >> > > concerns > > >> > > > > on > > >> > > > > > > the > > >> > > > > > > > > > > mailing thread. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Boyang > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Fri, May 8, 2020 at 5:26 PM Jun Rao < > > >> j...@confluent.io > > >> > > > > >> > > > > wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > >> Hi, Guozhang and Jason, > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> Thanks for the reply. A couple of more replies. > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> 102. Still not sure about this. How is the > > tombstone > > >> > issue > > >> > > > > > > addressed > > >> > > > > > > > > in > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> non-voter and the observer. They can die at any > > >> point > > >> > and > > >> > > > > > restart > > >> > > > > > > > at > > >> > > > > > > > > an > > >> > > > > > > > > > >> arbitrary later time, and the advancing of the > > >> > firstDirty > > >> > > > > offset > > >> > > > > > > and > > >> > > > > > > > > the > > >> > > > > > > > > > >> removal of the tombstone can happen > independently. > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> 106. I agree that it would be less confusing if > we > > >> used > > >> > > > > "epoch" > > >> > > > > > > > > instead > > >> > > > > > > > > > of > > >> > > > > > > > > > >> "leader epoch" consistently. > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> Jun > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> On Thu, May 7, 2020 at 4:04 PM Guozhang Wang < > > >> > > > > > wangg...@gmail.com> > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > Thanks Jun. Further replies are in-lined. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > On Mon, May 4, 2020 at 11:58 AM Jun Rao < > > >> > > j...@confluent.io > > >> > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > Hi, Guozhang, > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > Thanks for the reply. A few more replies > > inlined > > >> > > below. > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > On Sun, May 3, 2020 at 6:33 PM Guozhang Wang > < > > >> > > > > > > > wangg...@gmail.com> > > >> > > > > > > > > > >> wrote: > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > Hello Jun, > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > Thanks for your comments! I'm replying > inline > > >> > below: > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > On Fri, May 1, 2020 at 12:36 PM Jun Rao < > > >> > > > > j...@confluent.io > > >> > > > > > > > > >> > > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > 101. Bootstrapping related issues. > > >> > > > > > > > > > >> > > > > 101.1 Currently, we support auto broker > id > > >> > > > generation. > > >> > > > > > Is > > >> > > > > > > > this > > >> > > > > > > > > > >> > > supported > > >> > > > > > > > > > >> > > > > for bootstrap brokers? > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > The vote ids would just be the broker ids. > > >> > > > > > > "bootstrap.servers" > > >> > > > > > > > > > >> would be > > >> > > > > > > > > > >> > > > similar to what client configs have today, > > >> where > > >> > > > > > > > "quorum.voters" > > >> > > > > > > > > > >> would > > >> > > > > > > > > > >> > be > > >> > > > > > > > > > >> > > > pre-defined config values. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > My question was on the auto generated broker > > id. > > >> > > > > Currently, > > >> > > > > > > the > > >> > > > > > > > > > broker > > >> > > > > > > > > > >> > can > > >> > > > > > > > > > >> > > choose to have its broker Id auto generated. > > The > > >> > > > > generation > > >> > > > > > is > > >> > > > > > > > > done > > >> > > > > > > > > > >> > through > > >> > > > > > > > > > >> > > ZK to guarantee uniqueness. Without ZK, it's > > not > > >> > clear > > >> > > > how > > >> > > > > > the > > >> > > > > > > > > > broker > > >> > > > > > > > > > >> id > > >> > > > > > > > > > >> > is > > >> > > > > > > > > > >> > > auto generated. "quorum.voters" also can't be > > set > > >> > > > > statically > > >> > > > > > > if > > >> > > > > > > > > > broker > > >> > > > > > > > > > >> > ids > > >> > > > > > > > > > >> > > are auto generated. > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > Jason has explained some ideas that we've > > >> discussed > > >> > so > > >> > > > > far, > > >> > > > > > > the > > >> > > > > > > > > > >> reason we > > >> > > > > > > > > > >> > intentional did not include them so far is that > > we > > >> > feel > > >> > > it > > >> > > > > is > > >> > > > > > > > > out-side > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > scope of KIP-595. Under the umbrella of KIP-500 > > we > > >> > > should > > >> > > > > > > > definitely > > >> > > > > > > > > > >> > address them though. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > On the high-level, our belief is that "joining > a > > >> > quorum" > > >> > > > and > > >> > > > > > > > > "joining > > >> > > > > > > > > > >> (or > > >> > > > > > > > > > >> > more specifically, registering brokers in) the > > >> > cluster" > > >> > > > > would > > >> > > > > > be > > >> > > > > > > > > > >> > de-coupled a bit, where the former should be > > >> completed > > >> > > > > before > > >> > > > > > we > > >> > > > > > > > do > > >> > > > > > > > > > the > > >> > > > > > > > > > >> > latter. More specifically, assuming the quorum > is > > >> > > already > > >> > > > up > > >> > > > > > and > > >> > > > > > > > > > >> running, > > >> > > > > > > > > > >> > after the newly started broker found the leader > > of > > >> the > > >> > > > > quorum > > >> > > > > > it > > >> > > > > > > > can > > >> > > > > > > > > > >> send a > > >> > > > > > > > > > >> > specific RegisterBroker request including its > > >> > listener / > > >> > > > > > > protocol > > >> > > > > > > > / > > >> > > > > > > > > > etc, > > >> > > > > > > > > > >> > and upon handling it the leader can send back > the > > >> > > uniquely > > >> > > > > > > > generated > > >> > > > > > > > > > >> broker > > >> > > > > > > > > > >> > id to the new broker, while also executing the > > >> > > > > > "startNewBroker" > > >> > > > > > > > > > >> callback as > > >> > > > > > > > > > >> > the controller. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > 102. Log compaction. One weak spot of log > > >> > > compaction > > >> > > > > is > > >> > > > > > > for > > >> > > > > > > > > the > > >> > > > > > > > > > >> > > consumer > > >> > > > > > > > > > >> > > > to > > >> > > > > > > > > > >> > > > > deal with deletes. When a key is deleted, > > >> it's > > >> > > > > retained > > >> > > > > > > as a > > >> > > > > > > > > > >> > tombstone > > >> > > > > > > > > > >> > > > > first and then physically removed. If a > > >> client > > >> > > > misses > > >> > > > > > the > > >> > > > > > > > > > >> tombstone > > >> > > > > > > > > > >> > > > > (because it's physically removed), it may > > >> not be > > >> > > > able > > >> > > > > to > > >> > > > > > > > > update > > >> > > > > > > > > > >> its > > >> > > > > > > > > > >> > > > > metadata properly. The way we solve this > in > > >> > Kafka > > >> > > is > > >> > > > > > based > > >> > > > > > > > on > > >> > > > > > > > > a > > >> > > > > > > > > > >> > > > > configuration ( > > >> log.cleaner.delete.retention.ms) > > >> > > and > > >> > > > > we > > >> > > > > > > > > expect a > > >> > > > > > > > > > >> > > consumer > > >> > > > > > > > > > >> > > > > having seen an old key to finish reading > > the > > >> > > > deletion > > >> > > > > > > > > tombstone > > >> > > > > > > > > > >> > within > > >> > > > > > > > > > >> > > > that > > >> > > > > > > > > > >> > > > > time. There is no strong guarantee for > that > > >> > since > > >> > > a > > >> > > > > > broker > > >> > > > > > > > > could > > >> > > > > > > > > > >> be > > >> > > > > > > > > > >> > > down > > >> > > > > > > > > > >> > > > > for a long time. It would be better if we > > can > > >> > > have a > > >> > > > > > more > > >> > > > > > > > > > reliable > > >> > > > > > > > > > >> > way > > >> > > > > > > > > > >> > > of > > >> > > > > > > > > > >> > > > > dealing with deletes. > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > We propose to capture this in the > > >> > "FirstDirtyOffset" > > >> > > > > field > > >> > > > > > > of > > >> > > > > > > > > the > > >> > > > > > > > > > >> > quorum > > >> > > > > > > > > > >> > > > record fetch response: the offset is the > > >> maximum > > >> > > > offset > > >> > > > > > that > > >> > > > > > > > log > > >> > > > > > > > > > >> > > compaction > > >> > > > > > > > > > >> > > > has reached up to. If the follower has > > fetched > > >> > > beyond > > >> > > > > this > > >> > > > > > > > > offset > > >> > > > > > > > > > it > > >> > > > > > > > > > >> > > means > > >> > > > > > > > > > >> > > > itself is safe hence it has seen all > records > > >> up to > > >> > > > that > > >> > > > > > > > offset. > > >> > > > > > > > > On > > >> > > > > > > > > > >> > > getting > > >> > > > > > > > > > >> > > > the response, the follower can then decide > if > > >> its > > >> > > end > > >> > > > > > offset > > >> > > > > > > > > > >> actually > > >> > > > > > > > > > >> > > below > > >> > > > > > > > > > >> > > > that dirty offset (and hence may miss some > > >> > > > tombstones). > > >> > > > > If > > >> > > > > > > > > that's > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > case: > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > 1) Naively, it could re-bootstrap metadata > > log > > >> > from > > >> > > > the > > >> > > > > > very > > >> > > > > > > > > > >> beginning > > >> > > > > > > > > > >> > to > > >> > > > > > > > > > >> > > > catch up. > > >> > > > > > > > > > >> > > > 2) During that time, it would refrain > itself > > >> from > > >> > > > > > answering > > >> > > > > > > > > > >> > > MetadataRequest > > >> > > > > > > > > > >> > > > from any clients. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > I am not sure that the "FirstDirtyOffset" > field > > >> > fully > > >> > > > > > > addresses > > >> > > > > > > > > the > > >> > > > > > > > > > >> > issue. > > >> > > > > > > > > > >> > > Currently, the deletion tombstone is not > > removed > > >> > > > > immediately > > >> > > > > > > > > after a > > >> > > > > > > > > > >> > round > > >> > > > > > > > > > >> > > of cleaning. It's removed after a delay in a > > >> > > subsequent > > >> > > > > > round > > >> > > > > > > of > > >> > > > > > > > > > >> > cleaning. > > >> > > > > > > > > > >> > > Consider an example where a key insertion is > at > > >> > offset > > >> > > > 200 > > >> > > > > > > and a > > >> > > > > > > > > > >> deletion > > >> > > > > > > > > > >> > > tombstone of the key is at 400. Initially, > > >> > > > > FirstDirtyOffset > > >> > > > > > is > > >> > > > > > > > at > > >> > > > > > > > > > >> 300. A > > >> > > > > > > > > > >> > > follower/observer fetches from offset 0 and > > >> fetches > > >> > > the > > >> > > > > key > > >> > > > > > > at > > >> > > > > > > > > > offset > > >> > > > > > > > > > >> > 200. > > >> > > > > > > > > > >> > > A few rounds of cleaning happen. > > >> FirstDirtyOffset is > > >> > > at > > >> > > > > 500 > > >> > > > > > > and > > >> > > > > > > > > the > > >> > > > > > > > > > >> > > tombstone at 400 is physically removed. The > > >> > > > > > follower/observer > > >> > > > > > > > > > >> continues > > >> > > > > > > > > > >> > the > > >> > > > > > > > > > >> > > fetch, but misses offset 400. It catches all > > the > > >> way > > >> > > to > > >> > > > > > > > > > >> FirstDirtyOffset > > >> > > > > > > > > > >> > > and declares its metadata as ready. However, > > its > > >> > > > metadata > > >> > > > > > > could > > >> > > > > > > > be > > >> > > > > > > > > > >> stale > > >> > > > > > > > > > >> > > since it actually misses the deletion of the > > key. > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > Yeah good question, I should have put more > > >> details > > >> > in > > >> > > my > > >> > > > > > > > > explanation > > >> > > > > > > > > > >> :) > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > The idea is that we will adjust the log > > compaction > > >> for > > >> > > > this > > >> > > > > > raft > > >> > > > > > > > > based > > >> > > > > > > > > > >> > metadata log: before more details to be > > explained, > > >> > since > > >> > > > we > > >> > > > > > have > > >> > > > > > > > two > > >> > > > > > > > > > >> types > > >> > > > > > > > > > >> > of "watermarks" here, whereas in Kafka the > > >> watermark > > >> > > > > indicates > > >> > > > > > > > where > > >> > > > > > > > > > >> every > > >> > > > > > > > > > >> > replica have replicated up to and in Raft the > > >> > watermark > > >> > > > > > > indicates > > >> > > > > > > > > > where > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > majority of replicas (here only indicating > voters > > >> of > > >> > the > > >> > > > > > quorum, > > >> > > > > > > > not > > >> > > > > > > > > > >> > counting observers) have replicated up to, > let's > > >> call > > >> > > them > > >> > > > > > Kafka > > >> > > > > > > > > > >> watermark > > >> > > > > > > > > > >> > and Raft watermark. For this special log, we > > would > > >> > > > maintain > > >> > > > > > both > > >> > > > > > > > > > >> > watermarks. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > When log compacting on the leader, we would > only > > >> > compact > > >> > > > up > > >> > > > > to > > >> > > > > > > the > > >> > > > > > > > > > Kafka > > >> > > > > > > > > > >> > watermark, i.e. if there is at least one voter > > who > > >> > have > > >> > > > not > > >> > > > > > > > > replicated > > >> > > > > > > > > > >> an > > >> > > > > > > > > > >> > entry, it would not be compacted. The > > >> "dirty-offset" > > >> > is > > >> > > > the > > >> > > > > > > offset > > >> > > > > > > > > > that > > >> > > > > > > > > > >> > we've compacted up to and is communicated to > > other > > >> > > voters, > > >> > > > > and > > >> > > > > > > the > > >> > > > > > > > > > other > > >> > > > > > > > > > >> > voters would also compact up to this value --- > > i.e. > > >> > the > > >> > > > > > > difference > > >> > > > > > > > > > here > > >> > > > > > > > > > >> is > > >> > > > > > > > > > >> > that instead of letting each replica doing log > > >> > > compaction > > >> > > > > > > > > > independently, > > >> > > > > > > > > > >> > we'll have the leader to decide upon which > offset > > >> to > > >> > > > compact > > >> > > > > > to, > > >> > > > > > > > and > > >> > > > > > > > > > >> > propagate this value to others to follow, in a > > more > > >> > > > > > coordinated > > >> > > > > > > > > > manner. > > >> > > > > > > > > > >> > Also note when there are new voters joining the > > >> quorum > > >> > > who > > >> > > > > has > > >> > > > > > > not > > >> > > > > > > > > > >> > replicated up to the dirty-offset, of because > of > > >> other > > >> > > > > issues > > >> > > > > > > they > > >> > > > > > > > > > >> > truncated their logs to below the dirty-offset, > > >> they'd > > >> > > > have > > >> > > > > to > > >> > > > > > > > > > >> re-bootstrap > > >> > > > > > > > > > >> > from the beginning, and during this period of > > time > > >> the > > >> > > > > leader > > >> > > > > > > > > learned > > >> > > > > > > > > > >> about > > >> > > > > > > > > > >> > this lagging voter would not advance the > > watermark > > >> > (also > > >> > > > it > > >> > > > > > > would > > >> > > > > > > > > not > > >> > > > > > > > > > >> > decrement it), and hence not compacting either, > > >> until > > >> > > the > > >> > > > > > > voter(s) > > >> > > > > > > > > has > > >> > > > > > > > > > >> > caught up to that dirty-offset. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > So back to your example above, before the > > bootstrap > > >> > > voter > > >> > > > > gets > > >> > > > > > > to > > >> > > > > > > > > 300 > > >> > > > > > > > > > no > > >> > > > > > > > > > >> > log compaction would happen on the leader; and > > >> until > > >> > > later > > >> > > > > > when > > >> > > > > > > > the > > >> > > > > > > > > > >> voter > > >> > > > > > > > > > >> > have got to beyond 400 and hence replicated > that > > >> > > > tombstone, > > >> > > > > > the > > >> > > > > > > > log > > >> > > > > > > > > > >> > compaction would possibly get to that tombstone > > and > > >> > > remove > > >> > > > > it. > > >> > > > > > > Say > > >> > > > > > > > > > >> later it > > >> > > > > > > > > > >> > the leader's log compaction reaches 500, it can > > >> send > > >> > > this > > >> > > > > back > > >> > > > > > > to > > >> > > > > > > > > the > > >> > > > > > > > > > >> voter > > >> > > > > > > > > > >> > who can then also compact locally up to 500. > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > > 105. Quorum State: In addition to > VotedId, > > >> do we > > >> > > > need > > >> > > > > > the > > >> > > > > > > > > epoch > > >> > > > > > > > > > >> > > > > corresponding to VotedId? Over time, the > > same > > >> > > broker > > >> > > > > Id > > >> > > > > > > > could > > >> > > > > > > > > be > > >> > > > > > > > > > >> > voted > > >> > > > > > > > > > >> > > in > > >> > > > > > > > > > >> > > > > different generations with different > epoch. > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > Hmm, this is a good point. Originally I > think > > >> the > > >> > > > > > > > "LeaderEpoch" > > >> > > > > > > > > > >> field > > >> > > > > > > > > > >> > in > > >> > > > > > > > > > >> > > > that file is corresponding to the "latest > > known > > >> > > leader > > >> > > > > > > epoch", > > >> > > > > > > > > not > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > > "current leader epoch". For example, if the > > >> > current > > >> > > > > epoch > > >> > > > > > is > > >> > > > > > > > N, > > >> > > > > > > > > > and > > >> > > > > > > > > > >> > then > > >> > > > > > > > > > >> > > a > > >> > > > > > > > > > >> > > > vote-request with epoch N+1 is received and > > the > > >> > > voter > > >> > > > > > > granted > > >> > > > > > > > > the > > >> > > > > > > > > > >> vote > > >> > > > > > > > > > >> > > for > > >> > > > > > > > > > >> > > > it, then it means for this voter it knows > the > > >> > > "latest > > >> > > > > > epoch" > > >> > > > > > > > is > > >> > > > > > > > > N > > >> > > > > > > > > > + > > >> > > > > > > > > > >> 1 > > >> > > > > > > > > > >> > > > although it is unknown if that sending > > >> candidate > > >> > > will > > >> > > > > > indeed > > >> > > > > > > > > > become > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > new > > >> > > > > > > > > > >> > > > leader (which would only be notified via > > >> > > begin-quorum > > >> > > > > > > > request). > > >> > > > > > > > > > >> > However, > > >> > > > > > > > > > >> > > > when persisting the quorum state, we would > > >> encode > > >> > > > > > > leader-epoch > > >> > > > > > > > > to > > >> > > > > > > > > > >> N+1, > > >> > > > > > > > > > >> > > > while the leaderId to be the older leader. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > But now thinking about this a bit more, I > > feel > > >> we > > >> > > > should > > >> > > > > > use > > >> > > > > > > > two > > >> > > > > > > > > > >> > separate > > >> > > > > > > > > > >> > > > epochs, one for the "lates known" and one > for > > >> the > > >> > > > > > "current" > > >> > > > > > > to > > >> > > > > > > > > > pair > > >> > > > > > > > > > >> > with > > >> > > > > > > > > > >> > > > the leaderId. I will update the wiki page. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > Hmm, it's kind of weird to bump up the leader > > >> epoch > > >> > > > before > > >> > > > > > the > > >> > > > > > > > new > > >> > > > > > > > > > >> leader > > >> > > > > > > > > > >> > > is actually elected, right. > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > 106. "OFFSET_OUT_OF_RANGE: Used in the > > >> > > > > > FetchQuorumRecords > > >> > > > > > > > API > > >> > > > > > > > > to > > >> > > > > > > > > > >> > > indicate > > >> > > > > > > > > > >> > > > > that the follower has fetched from an > > invalid > > >> > > offset > > >> > > > > and > > >> > > > > > > > > should > > >> > > > > > > > > > >> > > truncate > > >> > > > > > > > > > >> > > > to > > >> > > > > > > > > > >> > > > > the offset/epoch indicated in the > > response." > > >> > > > Observers > > >> > > > > > > can't > > >> > > > > > > > > > >> truncate > > >> > > > > > > > > > >> > > > their > > >> > > > > > > > > > >> > > > > logs. What should they do with > > >> > > OFFSET_OUT_OF_RANGE? > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > I'm not sure if I understand your question? > > >> > > Observers > > >> > > > > > should > > >> > > > > > > > > still > > >> > > > > > > > > > >> be > > >> > > > > > > > > > >> > > able > > >> > > > > > > > > > >> > > > to truncate their logs as well. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > Hmm, I thought only the quorum nodes have > local > > >> logs > > >> > > and > > >> > > > > > > > observers > > >> > > > > > > > > > >> don't? > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > 107. "The leader will continue sending > > >> > > > BeginQuorumEpoch > > >> > > > > to > > >> > > > > > > > each > > >> > > > > > > > > > >> known > > >> > > > > > > > > > >> > > > voter > > >> > > > > > > > > > >> > > > > until it has received its endorsement." > If > > a > > >> > voter > > >> > > > is > > >> > > > > > down > > >> > > > > > > > > for a > > >> > > > > > > > > > >> long > > >> > > > > > > > > > >> > > > time, > > >> > > > > > > > > > >> > > > > sending BeginQuorumEpoch seems to add > > >> > unnecessary > > >> > > > > > > overhead. > > >> > > > > > > > > > >> > Similarly, > > >> > > > > > > > > > >> > > > if a > > >> > > > > > > > > > >> > > > > follower stops sending > FetchQuorumRecords, > > >> does > > >> > > the > > >> > > > > > leader > > >> > > > > > > > > keep > > >> > > > > > > > > > >> > sending > > >> > > > > > > > > > >> > > > > BeginQuorumEpoch? > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > Regarding BeginQuorumEpoch: that is a good > > >> point. > > >> > > The > > >> > > > > > > > > > >> > begin-quorum-epoch > > >> > > > > > > > > > >> > > > request is for voters to quickly get the > new > > >> > leader > > >> > > > > > > > information; > > >> > > > > > > > > > >> > however > > >> > > > > > > > > > >> > > > even if they do not get them they can still > > >> > > eventually > > >> > > > > > learn > > >> > > > > > > > > about > > >> > > > > > > > > > >> that > > >> > > > > > > > > > >> > > > from others via gossiping FindQuorum. I > think > > >> we > > >> > can > > >> > > > > > adjust > > >> > > > > > > > the > > >> > > > > > > > > > >> logic > > >> > > > > > > > > > >> > to > > >> > > > > > > > > > >> > > > e.g. exponential back-off or with a limited > > >> > > > num.retries. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > Regarding FetchQuorumRecords: if the > follower > > >> > sends > > >> > > > > > > > > > >> FetchQuorumRecords > > >> > > > > > > > > > >> > > > already, it means that follower already > knows > > >> that > > >> > > the > > >> > > > > > > broker > > >> > > > > > > > is > > >> > > > > > > > > > the > > >> > > > > > > > > > >> > > > leader, and hence we can stop retrying > > >> > > > BeginQuorumEpoch; > > >> > > > > > > > however > > >> > > > > > > > > > it > > >> > > > > > > > > > >> is > > >> > > > > > > > > > >> > > > possible that after a follower sends > > >> > > > FetchQuorumRecords > > >> > > > > > > > already, > > >> > > > > > > > > > >> > suddenly > > >> > > > > > > > > > >> > > > it stops send it (possibly because it > learned > > >> > about > > >> > > a > > >> > > > > > higher > > >> > > > > > > > > epoch > > >> > > > > > > > > > >> > > leader), > > >> > > > > > > > > > >> > > > and hence this broker may be a "zombie" > > leader > > >> and > > >> > > we > > >> > > > > > > propose > > >> > > > > > > > to > > >> > > > > > > > > > use > > >> > > > > > > > > > >> > the > > >> > > > > > > > > > >> > > > fetch.timeout to let the leader to try to > > >> verify > > >> > if > > >> > > it > > >> > > > > has > > >> > > > > > > > > already > > >> > > > > > > > > > >> been > > >> > > > > > > > > > >> > > > stale. > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > It just seems that we should handle these two > > >> cases > > >> > > in a > > >> > > > > > > > > consistent > > >> > > > > > > > > > >> way? > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > Yes I agree, on the leader's side, the > > >> > > > FetchQuorumRecords > > >> > > > > > > from a > > >> > > > > > > > > > >> follower > > >> > > > > > > > > > >> > could mean that we no longer needs to send > > >> > > > BeginQuorumEpoch > > >> > > > > > > > anymore > > >> > > > > > > > > > --- > > >> > > > > > > > > > >> and > > >> > > > > > > > > > >> > it is already part of our current > implementations > > >> in > > >> > > > > > > > > > >> > > > >> > > https://github.com/confluentinc/kafka/commits/kafka-raft > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > Thanks, > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > Jun > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > Jun > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > On Wed, Apr 29, 2020 at 8:56 PM Guozhang > > >> Wang < > > >> > > > > > > > > > wangg...@gmail.com > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > wrote: > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > Hello Leonard, > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > Thanks for your comments, I'm relying > in > > >> line > > >> > > > below: > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > On Wed, Apr 29, 2020 at 1:58 AM Wang > > >> (Leonard) > > >> > > Ge > > >> > > > < > > >> > > > > > > > > > >> > w...@confluent.io> > > >> > > > > > > > > > >> > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > Hi Kafka developers, > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > It's great to see this proposal and > it > > >> took > > >> > me > > >> > > > > some > > >> > > > > > > time > > >> > > > > > > > > to > > >> > > > > > > > > > >> > finish > > >> > > > > > > > > > >> > > > > > reading > > >> > > > > > > > > > >> > > > > > > it. > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > And I have the following questions > > about > > >> the > > >> > > > > > Proposal: > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > - How do we plan to test this > design > > >> to > > >> > > > ensure > > >> > > > > > its > > >> > > > > > > > > > >> > correctness? > > >> > > > > > > > > > >> > > Or > > >> > > > > > > > > > >> > > > > > more > > >> > > > > > > > > > >> > > > > > > broadly, how do we ensure that our > > new > > >> > > ‘pull’ > > >> > > > > > based > > >> > > > > > > > > model > > >> > > > > > > > > > >> is > > >> > > > > > > > > > >> > > > > > functional > > >> > > > > > > > > > >> > > > > > > and > > >> > > > > > > > > > >> > > > > > > correct given that it is different > > >> from > > >> > the > > >> > > > > > > original > > >> > > > > > > > > RAFT > > >> > > > > > > > > > >> > > > > > implementation > > >> > > > > > > > > > >> > > > > > > which has formal proof of > > correctness? > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > We have two planned verifications on > the > > >> > > > correctness > > >> > > > > > and > > >> > > > > > > > > > >> liveness > > >> > > > > > > > > > >> > of > > >> > > > > > > > > > >> > > > the > > >> > > > > > > > > > >> > > > > > design. One is via model verification > > >> (TLA+) > > >> > > > > > > > > > >> > > > > > > > >> > > > https://github.com/guozhangwang/kafka-specification > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > Another is via the concurrent > simulation > > >> tests > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://github.com/confluentinc/kafka/commit/5c0c054597d2d9f458cad0cad846b0671efa2d91 > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > - Have we considered any sensible > > >> defaults > > >> > > for > > >> > > > > the > > >> > > > > > > > > > >> > configuration, > > >> > > > > > > > > > >> > > > i.e. > > >> > > > > > > > > > >> > > > > > > all the election timeout, fetch > time > > >> out, > > >> > > > etc.? > > >> > > > > > Or > > >> > > > > > > we > > >> > > > > > > > > > want > > >> > > > > > > > > > >> to > > >> > > > > > > > > > >> > > > leave > > >> > > > > > > > > > >> > > > > > > this to > > >> > > > > > > > > > >> > > > > > > a later stage when we do the > > >> performance > > >> > > > > testing, > > >> > > > > > > > etc. > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > This is a good question, the reason we > > did > > >> not > > >> > > set > > >> > > > > any > > >> > > > > > > > > default > > >> > > > > > > > > > >> > values > > >> > > > > > > > > > >> > > > for > > >> > > > > > > > > > >> > > > > > the timeout configurations is that we > > >> think it > > >> > > may > > >> > > > > > take > > >> > > > > > > > some > > >> > > > > > > > > > >> > > > benchmarking > > >> > > > > > > > > > >> > > > > > experiments to get these defaults > right. > > >> Some > > >> > > > > > high-level > > >> > > > > > > > > > >> principles > > >> > > > > > > > > > >> > > to > > >> > > > > > > > > > >> > > > > > consider: 1) the fetch.timeout should > be > > >> > around > > >> > > > the > > >> > > > > > same > > >> > > > > > > > > scale > > >> > > > > > > > > > >> with > > >> > > > > > > > > > >> > > zk > > >> > > > > > > > > > >> > > > > > session timeout, which is now 18 > seconds > > by > > >> > > > default > > >> > > > > -- > > >> > > > > > > in > > >> > > > > > > > > > >> practice > > >> > > > > > > > > > >> > > > we've > > >> > > > > > > > > > >> > > > > > seen unstable networks having more than > > 10 > > >> > secs > > >> > > of > > >> > > > > > > > transient > > >> > > > > > > > > > >> > > > > connectivity, > > >> > > > > > > > > > >> > > > > > 2) the election.timeout, however, > should > > be > > >> > > > smaller > > >> > > > > > than > > >> > > > > > > > the > > >> > > > > > > > > > >> fetch > > >> > > > > > > > > > >> > > > > timeout > > >> > > > > > > > > > >> > > > > > as is also suggested as a practical > > >> > optimization > > >> > > > in > > >> > > > > > > > > > literature: > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> https://www.cl.cam.ac.uk/~ms705/pub/papers/2015-osr-raft.pdf > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > Some more discussions can be found > here: > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > >> > > > https://github.com/confluentinc/kafka/pull/301/files#r415420081 > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > - Have we considered piggybacking > > >> > > > > > > `BeginQuorumEpoch` > > >> > > > > > > > > with > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > ` > > >> > > > > > > > > > >> > > > > > > FetchQuorumRecords`? I might be > > >> missing > > >> > > > > something > > >> > > > > > > > > obvious > > >> > > > > > > > > > >> but > > >> > > > > > > > > > >> > I > > >> > > > > > > > > > >> > > am > > >> > > > > > > > > > >> > > > > > just > > >> > > > > > > > > > >> > > > > > > wondering why don’t we just use > the > > >> > > > > `FindQuorum` > > >> > > > > > > and > > >> > > > > > > > > > >> > > > > > > `FetchQuorumRecords` > > >> > > > > > > > > > >> > > > > > > APIs and remove the > > `BeginQuorumEpoch` > > >> > API? > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > Note that Begin/EndQuorumEpoch is sent > > from > > >> > > leader > > >> > > > > -> > > >> > > > > > > > other > > >> > > > > > > > > > >> voter > > >> > > > > > > > > > >> > > > > > followers, while FindQuorum / Fetch are > > >> sent > > >> > > from > > >> > > > > > > follower > > >> > > > > > > > > to > > >> > > > > > > > > > >> > leader. > > >> > > > > > > > > > >> > > > > > Arguably one can eventually realize the > > new > > >> > > leader > > >> > > > > and > > >> > > > > > > > epoch > > >> > > > > > > > > > via > > >> > > > > > > > > > >> > > > > gossiping > > >> > > > > > > > > > >> > > > > > FindQuorum, but that could in practice > > >> > require a > > >> > > > > long > > >> > > > > > > > delay. > > >> > > > > > > > > > >> > Having a > > >> > > > > > > > > > >> > > > > > leader -> other voters request helps > the > > >> new > > >> > > > leader > > >> > > > > > > epoch > > >> > > > > > > > to > > >> > > > > > > > > > be > > >> > > > > > > > > > >> > > > > propagated > > >> > > > > > > > > > >> > > > > > faster under a pull model. > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > - And about the > `FetchQuorumRecords` > > >> > > response > > >> > > > > > > schema, > > >> > > > > > > > > in > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > > > `Records` > > >> > > > > > > > > > >> > > > > > > field of the response, is it just > > one > > >> > > record > > >> > > > or > > >> > > > > > all > > >> > > > > > > > the > > >> > > > > > > > > > >> > records > > >> > > > > > > > > > >> > > > > > starting > > >> > > > > > > > > > >> > > > > > > from the FetchOffset? It seems a > lot > > >> more > > >> > > > > > efficient > > >> > > > > > > > if > > >> > > > > > > > > we > > >> > > > > > > > > > >> sent > > >> > > > > > > > > > >> > > all > > >> > > > > > > > > > >> > > > > the > > >> > > > > > > > > > >> > > > > > > records during the bootstrapping > of > > >> the > > >> > > > > brokers. > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > Yes the fetching is batched: > FetchOffset > > is > > >> > just > > >> > > > the > > >> > > > > > > > > starting > > >> > > > > > > > > > >> > offset > > >> > > > > > > > > > >> > > of > > >> > > > > > > > > > >> > > > > the > > >> > > > > > > > > > >> > > > > > batch of records. > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > - Regarding the disruptive broker > > >> issues, > > >> > > > does > > >> > > > > > our > > >> > > > > > > > pull > > >> > > > > > > > > > >> based > > >> > > > > > > > > > >> > > > model > > >> > > > > > > > > > >> > > > > > > suffer from it? If so, have we > > >> considered > > >> > > the > > >> > > > > > > > Pre-Vote > > >> > > > > > > > > > >> stage? > > >> > > > > > > > > > >> > If > > >> > > > > > > > > > >> > > > > not, > > >> > > > > > > > > > >> > > > > > > why? > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > The disruptive broker is stated in the > > >> > original > > >> > > > Raft > > >> > > > > > > paper > > >> > > > > > > > > > >> which is > > >> > > > > > > > > > >> > > the > > >> > > > > > > > > > >> > > > > > result of the push model design. Our > > >> analysis > > >> > > > showed > > >> > > > > > > that > > >> > > > > > > > > with > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > pull > > >> > > > > > > > > > >> > > > > > model it is no longer an issue. > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > Thanks a lot for putting this up, > and I > > >> hope > > >> > > > that > > >> > > > > my > > >> > > > > > > > > > questions > > >> > > > > > > > > > >> > can > > >> > > > > > > > > > >> > > be > > >> > > > > > > > > > >> > > > > of > > >> > > > > > > > > > >> > > > > > > some value to make this KIP better. > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > Hope to hear from you soon! > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > Best wishes, > > >> > > > > > > > > > >> > > > > > > Leonard > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > On Wed, Apr 29, 2020 at 1:46 AM Colin > > >> > McCabe < > > >> > > > > > > > > > >> cmcc...@apache.org > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > Hi Jason, > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > It's amazing to see this coming > > >> together > > >> > :) > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > I haven't had a chance to read in > > >> detail, > > >> > > but > > >> > > > I > > >> > > > > > read > > >> > > > > > > > the > > >> > > > > > > > > > >> > outline > > >> > > > > > > > > > >> > > > and > > >> > > > > > > > > > >> > > > > a > > >> > > > > > > > > > >> > > > > > > few > > >> > > > > > > > > > >> > > > > > > > things jumped out at me. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > First, for every epoch that is 32 > > bits > > >> > > rather > > >> > > > > than > > >> > > > > > > > 64, I > > >> > > > > > > > > > >> sort > > >> > > > > > > > > > >> > of > > >> > > > > > > > > > >> > > > > wonder > > >> > > > > > > > > > >> > > > > > > if > > >> > > > > > > > > > >> > > > > > > > that's a good long-term choice. I > > keep > > >> > > > reading > > >> > > > > > > about > > >> > > > > > > > > > stuff > > >> > > > > > > > > > >> > like > > >> > > > > > > > > > >> > > > > this: > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1277 > > >> > > > > > > > . > > >> > > > > > > > > > >> > > Obviously, > > >> > > > > > > > > > >> > > > > > that > > >> > > > > > > > > > >> > > > > > > > JIRA is about zxid, which > increments > > >> much > > >> > > > faster > > >> > > > > > > than > > >> > > > > > > > we > > >> > > > > > > > > > >> expect > > >> > > > > > > > > > >> > > > these > > >> > > > > > > > > > >> > > > > > > > leader epochs to, but it would > still > > be > > >> > good > > >> > > > to > > >> > > > > > see > > >> > > > > > > > some > > >> > > > > > > > > > >> rough > > >> > > > > > > > > > >> > > > > > > calculations > > >> > > > > > > > > > >> > > > > > > > about how long 32 bits (or really, > 31 > > >> > bits) > > >> > > > will > > >> > > > > > > last > > >> > > > > > > > us > > >> > > > > > > > > > in > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > > cases > > >> > > > > > > > > > >> > > > > > > where > > >> > > > > > > > > > >> > > > > > > > we're using it, and what the space > > >> savings > > >> > > > we're > > >> > > > > > > > getting > > >> > > > > > > > > > >> really > > >> > > > > > > > > > >> > > is. > > >> > > > > > > > > > >> > > > > It > > >> > > > > > > > > > >> > > > > > > > seems like in most cases the > tradeoff > > >> may > > >> > > not > > >> > > > be > > >> > > > > > > worth > > >> > > > > > > > > it? > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > Another thing I've been thinking > > about > > >> is > > >> > > how > > >> > > > we > > >> > > > > > do > > >> > > > > > > > > > >> > > > bootstrapping. I > > >> > > > > > > > > > >> > > > > > > > would prefer to be in a world where > > >> > > > formatting a > > >> > > > > > new > > >> > > > > > > > > Kafka > > >> > > > > > > > > > >> node > > >> > > > > > > > > > >> > > > was a > > >> > > > > > > > > > >> > > > > > > first > > >> > > > > > > > > > >> > > > > > > > class operation explicitly > initiated > > by > > >> > the > > >> > > > > admin, > > >> > > > > > > > > rather > > >> > > > > > > > > > >> than > > >> > > > > > > > > > >> > > > > > something > > >> > > > > > > > > > >> > > > > > > > that happened implicitly when you > > >> started > > >> > up > > >> > > > the > > >> > > > > > > > broker > > >> > > > > > > > > > and > > >> > > > > > > > > > >> > > things > > >> > > > > > > > > > >> > > > > > > "looked > > >> > > > > > > > > > >> > > > > > > > blank." > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > The first problem is that things > can > > >> "look > > >> > > > > blank" > > >> > > > > > > > > > >> accidentally > > >> > > > > > > > > > >> > if > > >> > > > > > > > > > >> > > > the > > >> > > > > > > > > > >> > > > > > > > storage system is having a bad day. > > >> > Clearly > > >> > > > in > > >> > > > > > the > > >> > > > > > > > > > non-Raft > > >> > > > > > > > > > >> > > world, > > >> > > > > > > > > > >> > > > > > this > > >> > > > > > > > > > >> > > > > > > > leads to data loss if the broker > that > > >> is > > >> > > > > > (re)started > > >> > > > > > > > > this > > >> > > > > > > > > > >> way > > >> > > > > > > > > > >> > was > > >> > > > > > > > > > >> > > > the > > >> > > > > > > > > > >> > > > > > > > leader for some partitions. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > The second problem is that we have > a > > >> bit > > >> > of > > >> > > a > > >> > > > > > > chicken > > >> > > > > > > > > and > > >> > > > > > > > > > >> egg > > >> > > > > > > > > > >> > > > problem > > >> > > > > > > > > > >> > > > > > > with > > >> > > > > > > > > > >> > > > > > > > certain configuration keys. For > > >> example, > > >> > > > maybe > > >> > > > > > you > > >> > > > > > > > want > > >> > > > > > > > > > to > > >> > > > > > > > > > >> > > > configure > > >> > > > > > > > > > >> > > > > > > some > > >> > > > > > > > > > >> > > > > > > > connection security settings in > your > > >> > > cluster, > > >> > > > > but > > >> > > > > > > you > > >> > > > > > > > > > don't > > >> > > > > > > > > > >> > want > > >> > > > > > > > > > >> > > > them > > >> > > > > > > > > > >> > > > > > to > > >> > > > > > > > > > >> > > > > > > > ever be stored in a plaintext > config > > >> file. > > >> > > > (For > > >> > > > > > > > > example, > > >> > > > > > > > > > >> SCRAM > > >> > > > > > > > > > >> > > > > > > passwords, > > >> > > > > > > > > > >> > > > > > > > etc.) You could use a broker API > to > > >> set > > >> > the > > >> > > > > > > > > > configuration, > > >> > > > > > > > > > >> but > > >> > > > > > > > > > >> > > > that > > >> > > > > > > > > > >> > > > > > > brings > > >> > > > > > > > > > >> > > > > > > > up the chicken and egg problem. > The > > >> > broker > > >> > > > > needs > > >> > > > > > to > > >> > > > > > > > be > > >> > > > > > > > > > >> > > configured > > >> > > > > > > > > > >> > > > to > > >> > > > > > > > > > >> > > > > > > know > > >> > > > > > > > > > >> > > > > > > > how to talk to you, but you need to > > >> > > configure > > >> > > > it > > >> > > > > > > > before > > >> > > > > > > > > > you > > >> > > > > > > > > > >> can > > >> > > > > > > > > > >> > > > talk > > >> > > > > > > > > > >> > > > > to > > >> > > > > > > > > > >> > > > > > > > it. Using an external secret > manager > > >> like > > >> > > > Vault > > >> > > > > > is > > >> > > > > > > > one > > >> > > > > > > > > > way > > >> > > > > > > > > > >> to > > >> > > > > > > > > > >> > > > solve > > >> > > > > > > > > > >> > > > > > > this, > > >> > > > > > > > > > >> > > > > > > > but not everyone uses an external > > >> secret > > >> > > > > manager. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > quorum.voters seems like a similar > > >> > > > configuration > > >> > > > > > > key. > > >> > > > > > > > > In > > >> > > > > > > > > > >> the > > >> > > > > > > > > > >> > > > current > > >> > > > > > > > > > >> > > > > > > KIP, > > >> > > > > > > > > > >> > > > > > > > this is only read if there is no > > other > > >> > > > > > configuration > > >> > > > > > > > > > >> specifying > > >> > > > > > > > > > >> > > the > > >> > > > > > > > > > >> > > > > > > quorum > > >> > > > > > > > > > >> > > > > > > > voter set. If we had a kafka.mkfs > > >> > command, > > >> > > we > > >> > > > > > > > wouldn't > > >> > > > > > > > > > need > > >> > > > > > > > > > >> > this > > >> > > > > > > > > > >> > > > key > > >> > > > > > > > > > >> > > > > > > > because we could assume that there > > was > > >> > > always > > >> > > > > > quorum > > >> > > > > > > > > > >> > information > > >> > > > > > > > > > >> > > > > stored > > >> > > > > > > > > > >> > > > > > > > locally. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > best, > > >> > > > > > > > > > >> > > > > > > > Colin > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Thu, Apr 16, 2020, at 16:44, > Jason > > >> > > > Gustafson > > >> > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Hi All, > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > I'd like to start a discussion on > > >> > KIP-595: > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum > > >> > > > > > > > > > >> > > > > > > > . > > >> > > > > > > > > > >> > > > > > > > > This proposal specifies a Raft > > >> protocol > > >> > to > > >> > > > > > > > ultimately > > >> > > > > > > > > > >> replace > > >> > > > > > > > > > >> > > > > > Zookeeper > > >> > > > > > > > > > >> > > > > > > > > as > > >> > > > > > > > > > >> > > > > > > > > documented in KIP-500. Please > take > > a > > >> > look > > >> > > > and > > >> > > > > > > share > > >> > > > > > > > > your > > >> > > > > > > > > > >> > > > thoughts. > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > A few minor notes to set the > stage > > a > > >> > > little > > >> > > > > bit: > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > - This KIP does not specify the > > >> > structure > > >> > > of > > >> > > > > the > > >> > > > > > > > > > messages > > >> > > > > > > > > > >> > used > > >> > > > > > > > > > >> > > to > > >> > > > > > > > > > >> > > > > > > > represent > > >> > > > > > > > > > >> > > > > > > > > metadata in Kafka, nor does it > > >> specify > > >> > the > > >> > > > > > > internal > > >> > > > > > > > > API > > >> > > > > > > > > > >> that > > >> > > > > > > > > > >> > > will > > >> > > > > > > > > > >> > > > > be > > >> > > > > > > > > > >> > > > > > > used > > >> > > > > > > > > > >> > > > > > > > > by the controller. Expect these > to > > >> come > > >> > in > > >> > > > > later > > >> > > > > > > > > > >> proposals. > > >> > > > > > > > > > >> > > Here > > >> > > > > > > > > > >> > > > we > > >> > > > > > > > > > >> > > > > > are > > >> > > > > > > > > > >> > > > > > > > > primarily concerned with the > > >> replication > > >> > > > > > protocol > > >> > > > > > > > and > > >> > > > > > > > > > >> basic > > >> > > > > > > > > > >> > > > > > operational > > >> > > > > > > > > > >> > > > > > > > > mechanics. > > >> > > > > > > > > > >> > > > > > > > > - We expect many details to > change > > >> as we > > >> > > get > > >> > > > > > > closer > > >> > > > > > > > to > > >> > > > > > > > > > >> > > > integration > > >> > > > > > > > > > >> > > > > > with > > >> > > > > > > > > > >> > > > > > > > > the controller. Any changes we > make > > >> will > > >> > > be > > >> > > > > made > > >> > > > > > > > > either > > >> > > > > > > > > > as > > >> > > > > > > > > > >> > > > > amendments > > >> > > > > > > > > > >> > > > > > > to > > >> > > > > > > > > > >> > > > > > > > > this KIP or, in the case of > larger > > >> > > changes, > > >> > > > as > > >> > > > > > new > > >> > > > > > > > > > >> proposals. > > >> > > > > > > > > > >> > > > > > > > > - We have a prototype > > implementation > > >> > > which I > > >> > > > > > will > > >> > > > > > > > put > > >> > > > > > > > > > >> online > > >> > > > > > > > > > >> > > > within > > >> > > > > > > > > > >> > > > > > the > > >> > > > > > > > > > >> > > > > > > > > next week which may help in > > >> > understanding > > >> > > > some > > >> > > > > > > > > details. > > >> > > > > > > > > > It > > >> > > > > > > > > > >> > has > > >> > > > > > > > > > >> > > > > > > diverged a > > >> > > > > > > > > > >> > > > > > > > > little bit from our proposal, so > I > > am > > >> > > > taking a > > >> > > > > > > > little > > >> > > > > > > > > > >> time to > > >> > > > > > > > > > >> > > > bring > > >> > > > > > > > > > >> > > > > > it > > >> > > > > > > > > > >> > > > > > > in > > >> > > > > > > > > > >> > > > > > > > > line. I'll post an update to this > > >> thread > > >> > > > when > > >> > > > > it > > >> > > > > > > is > > >> > > > > > > > > > >> available > > >> > > > > > > > > > >> > > for > > >> > > > > > > > > > >> > > > > > > review. > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Finally, I want to mention that > > this > > >> > > > proposal > > >> > > > > > was > > >> > > > > > > > > > drafted > > >> > > > > > > > > > >> by > > >> > > > > > > > > > >> > > > > myself, > > >> > > > > > > > > > >> > > > > > > > Boyang > > >> > > > > > > > > > >> > > > > > > > > Chen, and Guozhang Wang. > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Thanks, > > >> > > > > > > > > > >> > > > > > > > > Jason > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > -- > > >> > > > > > > > > > >> > > > > > > Leonard Ge > > >> > > > > > > > > > >> > > > > > > Software Engineer Intern - Confluent > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > -- > > >> > > > > > > > > > >> > > > > > -- Guozhang > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > -- > > >> > > > > > > > > > >> > > > -- Guozhang > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > -- > > >> > > > > > > > > > >> > -- Guozhang > > >> > > > > > > > > > >> > > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -- > > >> > > > > > > > > -- Guozhang > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > -- Guozhang > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
