Yeah, the reason why we want to deprecate the auto create functionality is that it happens when a metadata request is done instead of when a write operation happens. So, there's no reason to differentiate between the two.
Ismael On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto <o...@wikimedia.org> wrote: > Ah, I just realized that as proposed this is only for the Java consumer > client, correct? Would it be possible to make this a broker config, like > the current one? Something like: > > auto.create.topics.enable=true # allow both producer and consumer to create > auto.create.topics.enable=consumer # allow only consumer to create > auto.create.topics.enable=producer # allow only producer to create > auto.create.topics.enable=false # deny any auto topic creation > > Perhaps the broker doesn’t differentiate between the type of client > connection. If not, I guess this wouldn’t be possible. > > > > On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto <o...@wikimedia.org> wrote: > > > Yup :) > > > > On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > >> Andrew, one question: you are relying on auto topic creation for the > >> producer and that's why you can't just disable it? > >> > >> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <ism...@juma.me.uk> wrote: > >> > >> > Thanks for sharing Andrew! > >> > > >> > Ismael > >> > > >> > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <o...@wikimedia.org> > wrote: > >> > > >> >> We recently had a pretty serious Kafka outage > >> >> < > >> >> > >> > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary > >> >> > > >> >> caused by a bug in one of our consumers that caused it to create new > >> >> topics > >> >> in an infinite loop AKA a topic bomb! Having consumers restricted > from > >> >> creating topics would have prevented this for us. > >> >> > >> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ism...@juma.me.uk> > wrote: > >> >> > >> >> > Generally, I think positive configs (`allow` instead of `suppress`) > >> are > >> >> > easier to understand. > >> >> > > >> >> > Ismael > >> >> > > >> >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax < > >> matth...@confluent.io > >> >> > > >> >> > wrote: > >> >> > > >> >> > > Thanks for the summary! > >> >> > > > >> >> > > We might want to add a diagram/table to the docs when we add this > >> >> > > feature (with whatever config name we choose) to explain how > broker > >> >> > > config `auto.create.topics.enable` and the consumer config work > >> >> together. > >> >> > > > >> >> > > I think both options are equally easy to understand. "allow" > means > >> >> > > follow the broker config, while "suppress" implies ignore the > >> broker > >> >> > > config and don't auto-create. > >> >> > > > >> >> > > > >> >> > > -Matthias > >> >> > > > >> >> > > > >> >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote: > >> >> > > > *"suppress" is the opposite of "allow", so > >> >> > > > setting suppress.auto.create.topics=false would mean that we do > >> >> _not_ > >> >> > > allow > >> >> > > > auto topic creation; when set to true, the server configuration > >> will > >> >> > > > determine whether we allow automatic creation or not.* > >> >> > > > > >> >> > > > Sorry, I meant suppress.auto.create.topics=true above to > disallow > >> >> auto > >> >> > > > topic creation. > >> >> > > > > >> >> > > > > >> >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah < > >> dhru...@confluent.io > >> >> > > >> >> > > wrote: > >> >> > > > > >> >> > > >> To be clear, we will allow auto topic creation only when > server > >> >> config > >> >> > > >> auto.create.topics.enable=true and consumer config > >> >> > > >> allow.auto.create.topics=true; when either is false, we would > >> not > >> >> > create > >> >> > > >> the topic if it does not exist. > >> >> > > >> > >> >> > > >> "suppress" is the opposite of "allow", so > >> >> > > >> setting suppress.auto.create.topics=false would mean that we > do > >> >> _not_ > >> >> > > allow > >> >> > > >> auto topic creation; when set to true, the server > configuration > >> >> will > >> >> > > >> determine whether we allow automatic creation or not. > >> >> > > >> > >> >> > > >> I think "allow" is easier to understand but I am open to > >> >> suggestions. > >> >> > > >> > >> >> > > >> - Dhruvil > >> >> > > >> > >> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner < > >> >> > > >> brandon.kirch...@gmail.com> wrote: > >> >> > > >> > >> >> > > >>> “allow=false” seems a bit more intuitive to me than > >> >> “suppress=false” > >> >> > > >>> > >> >> > > >>> Brandon > >> >> > > >>> > >> >> > > >>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhih...@gmail.com> > >> wrote: > >> >> > > >>>> > >> >> > > >>>> We may also consider : > >> >> > > >>>> > >> >> > > >>>> "suppress.auto.topic.creation" > >> >> > > >>>> > >> >> > > >>>> or > >> >> > > >>>> > >> >> > > >>>> "allow.auto.topic.creation" > >> >> > > >>>> > >> >> > > >>>> w.r.t. suppress or allow, I don't have strong opinion > either. > >> >> It's > >> >> > > just > >> >> > > >>> a > >> >> > > >>>> matter of choosing the proper default value. > >> >> > > >>>> > >> >> > > >>>> Cheers > >> >> > > >>>> > >> >> > > >>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah < > >> >> dhru...@confluent.io > >> >> > > > >> >> > > >>> wrote: > >> >> > > >>>>> > >> >> > > >>>>> Hi Matthias, > >> >> > > >>>>> > >> >> > > >>>>> Do you mean something like "suppress.auto.create.topic"? I > am > >> >> > leaning > >> >> > > >>> a bit > >> >> > > >>>>> towards "allow.auto.create.topics" but I don't have a > strong > >> >> > > preference > >> >> > > >>>>> either. Let's wait to hear if anyone else has an opinion on > >> >> this. > >> >> > > >>>>> > >> >> > > >>>>> Thanks, > >> >> > > >>>>> Dhruvil > >> >> > > >>>>> > >> >> > > >>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax < > >> >> > > matth...@confluent.io > >> >> > > >>>> > >> >> > > >>>>> wrote: > >> >> > > >>>>> > >> >> > > >>>>>> Thanks for the KIP Dhruvil! > >> >> > > >>>>>> > >> >> > > >>>>>> I agree with Jason's comment. An alternative might be to > use > >> >> > > >>> "suppress" > >> >> > > >>>>>> what would revert the logic of "allow". Not sure which one > >> is > >> >> more > >> >> > > >>>>>> intuitive and I am fine with both (no personal > preference). > >> >> Just > >> >> > > >>> wanted > >> >> > > >>>>>> to mention it as an alternative. > >> >> > > >>>>>> > >> >> > > >>>>>> Don't have any further comments/question so far. > >> >> > > >>>>>> > >> >> > > >>>>>> > >> >> > > >>>>>> -Matthias > >> >> > > >>>>>> > >> >> > > >>>>>> > >> >> > > >>>>>> > >> >> > > >>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote: > >> >> > > >>>>>>> Hey Dhruvil, > >> >> > > >>>>>>> > >> >> > > >>>>>>> I would suggest using the verb "allow" rather than > "enable. > >> >> The > >> >> > > >>>>> consumer > >> >> > > >>>>>>> cannot enable auto topic creation because it is > configured > >> on > >> >> the > >> >> > > >>>>> broker. > >> >> > > >>>>>>> All it can do is prevent it from happening if it is > >> enabled. > >> >> > > >>>>>>> > >> >> > > >>>>>>> -Jason > >> >> > > >>>>>>> > >> >> > > >>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah < > >> >> > > dhru...@confluent.io> > >> >> > > >>>>>> wrote: > >> >> > > >>>>>>> > >> >> > > >>>>>>>> Hi, > >> >> > > >>>>>>>> > >> >> > > >>>>>>>> I would like to start discussion on KIP-361 that > proposes > >> we > >> >> > add a > >> >> > > >>>>>> consumer > >> >> > > >>>>>>>> configuration to disable auto topic creation. > >> >> > > >>>>>>>> > >> >> > > >>>>>>>> Link to the KIP: > >> >> > > >>>>>>>> > >> >> > > >>>>>> > >> >> > > >>>>> > >> >> > > >>> > >> >> > > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+ > >> >> > > >>>>>>>> Configuration+to+Disable+Auto+Topic+Creation > >> >> > > >>>>>>>> > >> >> > > >>>>>>>> Suggestions and feedback are welcome! > >> >> > > >>>>>>>> > >> >> > > >>>>>>>> Thanks, > >> >> > > >>>>>>>> Dhruvil > >> >> > > >>>>>>>> > >> >> > > >>>>>>> > >> >> > > >>>>>> > >> >> > > >>>>>> > >> >> > > >>>>> > >> >> > > >>> > >> >> > > >> > >> >> > > > > >> >> > > > >> >> > > > >> >> > > >> >> > >> > > >> > > >
