Matthias J. Sax created KAFKA-8821:
--------------------------------------

             Summary: Avoid pattern subscription to allow for stricter ACL 
settings
                 Key: KAFKA-8821
                 URL: https://issues.apache.org/jira/browse/KAFKA-8821
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax


To avoid triggering auto topic creation (if `auto.create.topic.enable=true` on 
the brokers), Kafka Streams uses consumer pattern subscription. For this case, 
the consumer requests all metadata from the brokers and does client side 
filtering.

However, if users want to set ACL to restrict a Kafka Streams application, this 
may results in broker side ERROR logs that some metadata cannot be provided. 
The only way to avoid those broker side ERROR logs is to grant corresponding 
permissions.

As of 2.3 release it's possible to disable auto topic creation client side (via 
https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should use 
this new feature (note, that broker version 0.11 is required) to allow users to 
set strict ACLs without getting flooded with ERROR logs on the broker.

The proposal is that by default Kafka Streams disables auto-topic create client 
side (optimistically) and uses regular subscription (not pattern subscription). 
If an older broker is used, users need to explicitly enable 
`allow.auto.create.topic` client side. If we detect this setting, we switch 
back to pattern based subscription.

If users don't enable auto topic create client side and run with an older 
broker, we would just rethrow the exception to the user, adding some context 
information on how to fix the issue. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Reply via email to