With the KIP-4 create topic schema voted and passed and a PR available
upstream. I wanted to discuss moving the auto topic creation from the
broker side to the client side (KAFKA-2410
<https://issues.apache.org/jira/browse/KAFKA-2410>).
This change has many benefits
- Remove the need for failed messages until a topic is created
- Client can define the auto create parameters instead of a global
cluster setting
- Errors can be communicated back to the client more clearly
Overall auto create is not my favorite feature, since topic creation is a
highly critical piece for Kafka, and with authorization added it becomes
even more involved. When creating a topic a user needs:
- The access to create topics
- To set the correct partition count and replication factor for their
use case
- To set who has access to the topic
- Knowledge of how a new topic may impact regex consumers or mirrormaker
Often I find use cases that look like they need auto topic creation, can
often be handled with a few pre made topics. That said, we still should
support the feature for the cases that need it (mirrormaker, streams).
The question is how we should expose auto topic creation in the client. A
few options are:
- Add configs like the broker configs today, and let the client
automatically create the topics if enabled
- Both producer and consumer?
- Throw an error to the user and let them use a separate AdminClient
(KIP-4) api to create the topic
- Throw an error to the user and add a create api to the producer so
they can easily handle by creating a topic
I am leaning towards the last 2 options but wanted to get some others
thoughts on the matter. Especially if you have use cases that use auto
topic creation today.
Thanks,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
