Hey Grant, I'm also inclined to the latter two options you've suggested (and probably more to the one which uses AdminClient since it encourages separation of privileges). Raising an exception gives the user the most control over how to handle unknown topics. One of the problems that we currently have with the clients is that we retry silently on unknown topics under the expectation that they will eventually be created (automatically or not). This makes it difficult to detect misconfiguration without looking for warnings in the logs. This problem is compounded if the client isn't authorized to the topic since then we don't actually know if the topic exists or not and whether it is reasonable to keep retrying. I think a safer default would be to treat unknown topics as fatal errors. Of course, this is slightly orthogonal to the question of autocreation. We could change the behavior to raise an exception only if autocreation is not enabled, but I'd rather push users to creating topics with the AdminClient since it saves us from needing to add a bunch of default topic configuration options to the client configurations.
Thanks, Jason On Tue, Jun 28, 2016 at 9:26 AM, Grant Henke <ghe...@cloudera.com> wrote: > With the KIP-4 create topic schema voted and passed and a PR available > upstream. I wanted to discuss moving the auto topic creation from the > broker side to the client side (KAFKA-2410 > <https://issues.apache.org/jira/browse/KAFKA-2410>). > > This change has many benefits > > - Remove the need for failed messages until a topic is created > - Client can define the auto create parameters instead of a global > cluster setting > - Errors can be communicated back to the client more clearly > > Overall auto create is not my favorite feature, since topic creation is a > highly critical piece for Kafka, and with authorization added it becomes > even more involved. When creating a topic a user needs: > > - The access to create topics > - To set the correct partition count and replication factor for their > use case > - To set who has access to the topic > - Knowledge of how a new topic may impact regex consumers or mirrormaker > > Often I find use cases that look like they need auto topic creation, can > often be handled with a few pre made topics. That said, we still should > support the feature for the cases that need it (mirrormaker, streams). > > The question is how we should expose auto topic creation in the client. A > few options are: > > - Add configs like the broker configs today, and let the client > automatically create the topics if enabled > - Both producer and consumer? > - Throw an error to the user and let them use a separate AdminClient > (KIP-4) api to create the topic > - Throw an error to the user and add a create api to the producer so > they can easily handle by creating a topic > > I am leaning towards the last 2 options but wanted to get some others > thoughts on the matter. Especially if you have use cases that use auto > topic creation today. > > Thanks, > Grant > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
