Hi Colin, Thanks for your interest in this. I agree with you, this change could break compatibility. However, changing the source principal is non trivial in this case. In fact, here the problem is not in the internal topic creation - which succeeds - but in the two subsequent LeaderAndIsr and UpdateMetadata requests.
When a consumer tries to consume for the first time, the creation of internal topic completes, zk-nodes are filled with the necessary metadata, and this triggers a ZkPartitionStateMachine (PartitionStateMachine.scala) update which, in turn, makes the ControllerChannelManager (ControllerChannelManager.scala) send LeaderAndIsr and UpdateMetadata requests to the brokers; (I can be wrong, but I believe that this requests are already being executed with broker principal). These requests fail because we authorize the cluster operation there, so the __consumer_offsets topic remains in a bad state. Is there a reason to not authorize the operation for find coordinator requests as well? Thanks, Paolo On Mon, 6 Apr 2020 at 23:58, Colin McCabe <cmcc...@apache.org> wrote: > Hi Paolo, > > Thanks for finding this issue. > > Unfortunately, you certainly can't add a new permission requirement to an > existing RPC without breaking compatibility. So the current solution in > the PR will not work. However, you should be able to have the broker > create the topic using its own principal rather than the caller's. > Basically the equivalent of a doAs block (I forget how we do this exactly, > but we do have some way of doing it). > > best, > Colin > > > On Mon, Apr 6, 2020, at 02:56, Paolo Moriello wrote: > > Hello everybody, > > > > I've opened a Jira to fix a bug on creation of internal topics. This > > happens when the topics are created under insufficient ACLs: eg. > > __consumer_offset is created but subsequent updateMetadata and leaderIsr > > requests fail; the topic is than in an inconsistent state and it is > > impossible to consume. > > > > Jira: https://issues.apache.org/jira/browse/KAFKA-9806 > > > > A simple fix to solve this problem is to authorize the cluster operation > > before creating these topics. I've submitted a PR with the fix: > > https://github.com/apache/kafka/pull/8415 > > > > Please take a look and let me know if you have any feedback. > > Thanks, > > Paolo > > >
