Thanks Colin for the suggestions! I have added metrics to track the number of total messages being forwarded as a more generic monitoring. Right now the list of metrics are:
- num-client-forwarding-to-controller-rate - num-client-fowarding-to-controller-count - num-messages-redirected-rate - num-messages-redirected-count - request-forwarding-authorization-fail-count (Optional) Let me know if we need to add more monitoring tags. Boyang On Fri, Apr 17, 2020 at 1:16 PM Colin McCabe <cmcc...@apache.org> wrote: > On Thu, Apr 16, 2020, at 12:33, Jose Garcia Sancio wrote: > > Hi Boyang, > > > > Thanks for the KIP. The KIP looks good. I have a few questions and > comments. > > > > > As part of the KIP-500 > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > > > initiative, we need to build a bridge release version of Kafka that > > could > > isolate the direct Zookeeper write access only to the controller. > > > > This may be outside the scope of this KIP but I am trying to understand > how > > this is going to be used to implement KIP-500. My understanding is that > the > > Admin client discovers the controller by performing a > > Metadata{Request,Response} round trip. The response of this request > > includes the id of the controller. Based on my understanding of the > > KIP-500, this architecture will need to change. > > Hi Jose, > > Thanks for the questions. > > We're not proposing to change the MetadataRequest / MetadataResponse. It > will still work the same way that it always has from the perspective of > clients. We have to do this to maintain compatibility, I think. > > This does mean that controllers and brokers will occupy the same ID > namespace, even when running in standalone controller mode. I think that's > OK, though. > > > For example the controller > > will not necessarily be a broker in which case the id may not correlate > to > > a broker id. Is the expectation that the Kafka Controller Quorum (as > > defined in KIP-500) will push this new connection information to all of > the > > brokers? Will the Kafka Controller Quorum expose and implement all of the > > RPCs being redirected in this KIP and the ones that are currently routed > to > > the controller? These include: > > > > ListPartitionReassignment > > AlterPartitionReassignment > > ElectLeaders > > CreatePartitions > > DeleteTopics > > CreateTopics > > Yes, the KIP-500 controller (quorum) will implement all of those RPCs. > > > > > > AUTHORIZATION_FAILED if the inter-broker verification failed. > > > > The rest of the document mentions CLUSTER_AUTHORIZATION_FAILED. > > > > > For CLUSTER_AUTHORIZATION_FAILED, this indicates an internal error for > > broker security setup which has nothing to do with the client, so we have > > no other way but returning an UNKNOWN_SERVER_ERROR to the admin client. > > > > I don't understand this. I think I don't understand this because it is > not > > clear to me who, how and when authorization is going to work when using > > Envelopre{Request,Response}. Can you please add a section that explains > how > > authorization works when envelopes are involved? > > The only way a broker would fail to authorize itself to another broker is > if the ACLs were incorrectly set up for the cluster. This would cause > other problems beyond just making forwarding not work, though. > > This isn't really related to envelopes -- any time a broker makes an RPC > to another broker, in theory the ACLs could be totally screwed up and we > could get denied. It clearly means that administrator made a mistake, but > we still have to handle the case somehow. > > best, > Colin >
