How about this: We create a configuration key called early.start.listeners which contains a list of listener names. If this is not specified, its value defaults to just the controller listeners. Optionally, other listeners can be added too.
If super.users contains any user names, early start listeners will start immediately. In the beginning they only authorize users that are in super.users. All other listeners receive a new error code, AUTHORIZER_NOT_READY_ERROR. If super.users does not contain any user names, then early start listeners will not be treated differently than other listeners. This will allow the controller listeners to get started immediately if the broker user is in super.users, which will speed up startup. It also will be useful for breaking chicken/egg cycles like needing to pull the SCRAM metadata to authorize pulling the SCRAM metadata. There are still a few use cases where super.users won't be required, but it may be useful in many cases to have this early start functionality. Leaving aside the preceding discussion, do you agree with starting up all endpoints (including non-early start ones) once we load a metadata snapshot? How feasible would it be for us to get a callback from the Raft layer the first time we caught up to the last stable offset? (we only want the callback the first time, not any other time). (I think the metadata shell also would want something like this, at least as an option). best, Colin On Tue, Jan 25, 2022, at 13:34, Jason Gustafson wrote: > Hi Colin, > > Thanks for the writeup. I had one question about bootstrapping. For the > brokers, I understand that listener startup is delayed until the Authorizer > is ready. However, I was not very clear how this would work for the > controller listeners. We may need them to startup before the metadata log > is ready so that a quorum can be established (as noted in the KIP). This > works fine if we assume that the controller principals are among > `super.users`. For requests forwarded from brokers, on the other hand, we > need to ensure the ACLs have been loaded properly before we begin > authorizing. The problem is that we currently use the same listener for > quorum requests and for forwarded requests. So my question is how does the > Authorizer communicate to the controller when it is safe to begin > authorizing different request types? > > There are a couple ways I can see this working. First, we could allow the > user to configure the listener used for forwarded requests separately. That > would work with the existing `Authorizer.start` API. Alternatively, perhaps > we could modify `Authorizer.start` to work with something more granular > than `EndPoint`. This would allow the controller to begin accepting > requests from the other quorum members before it is ready to authorize > forwarded requests from clients. Then we would need some way to let > brokers know when the controller is ready to accept these forwarded > requests (e.g. through an error code in the `Envelope` response). > > What do you think? > > Thanks, > Jason > > > > > > > On Wed, Jan 12, 2022 at 12:57 PM David Arthur > <david.art...@confluent.io.invalid> wrote: > >> +1 binding, thanks Colin! >> >> On Mon, Dec 13, 2021 at 7:47 PM Colin McCabe <cmcc...@apache.org> wrote: >> >> > Hi all, >> > >> > I'd like to start the vote on KIP-801: Implement an Authorizer that >> stores >> > metadata in __cluster_metadata >> > >> > The KIP is here: https://cwiki.apache.org/confluence/x/h5KqCw >> > >> > The original DISCUSS thread is here: >> > >> > https://lists.apache.org/thread/3d5o7h17ztjztjhblx4fln0wbbs1rmdq >> > >> > Please take a look and vote if you can. >> > >> > best, >> > Colin >> > >> >> >> -- >> -David >>
