What is the status of this KIP? I think you can start a VOTE Dhruvil.
-Matthias On 8/23/18 9:52 AM, Ismael Juma wrote: > Yeah, the reason why we want to deprecate the auto create functionality is > that it happens when a metadata request is done instead of when a write > operation happens. So, there's no reason to differentiate between the two. > > Ismael > > On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto <o...@wikimedia.org> wrote: > >> Ah, I just realized that as proposed this is only for the Java consumer >> client, correct? Would it be possible to make this a broker config, like >> the current one? Something like: >> >> auto.create.topics.enable=true # allow both producer and consumer to create >> auto.create.topics.enable=consumer # allow only consumer to create >> auto.create.topics.enable=producer # allow only producer to create >> auto.create.topics.enable=false # deny any auto topic creation >> >> Perhaps the broker doesn’t differentiate between the type of client >> connection. If not, I guess this wouldn’t be possible. >> >> >> >> On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto <o...@wikimedia.org> wrote: >> >>> Yup :) >>> >>> On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <ism...@juma.me.uk> wrote: >>> >>>> Andrew, one question: you are relying on auto topic creation for the >>>> producer and that's why you can't just disable it? >>>> >>>> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <ism...@juma.me.uk> wrote: >>>> >>>>> Thanks for sharing Andrew! >>>>> >>>>> Ismael >>>>> >>>>> On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <o...@wikimedia.org> >> wrote: >>>>> >>>>>> We recently had a pretty serious Kafka outage >>>>>> < >>>>>> >>>> >> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary >>>>>>> >>>>>> caused by a bug in one of our consumers that caused it to create new >>>>>> topics >>>>>> in an infinite loop AKA a topic bomb! Having consumers restricted >> from >>>>>> creating topics would have prevented this for us. >>>>>> >>>>>> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ism...@juma.me.uk> >> wrote: >>>>>> >>>>>>> Generally, I think positive configs (`allow` instead of `suppress`) >>>> are >>>>>>> easier to understand. >>>>>>> >>>>>>> Ismael >>>>>>> >>>>>>> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax < >>>> matth...@confluent.io >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the summary! >>>>>>>> >>>>>>>> We might want to add a diagram/table to the docs when we add this >>>>>>>> feature (with whatever config name we choose) to explain how >> broker >>>>>>>> config `auto.create.topics.enable` and the consumer config work >>>>>> together. >>>>>>>> >>>>>>>> I think both options are equally easy to understand. "allow" >> means >>>>>>>> follow the broker config, while "suppress" implies ignore the >>>> broker >>>>>>>> config and don't auto-create. >>>>>>>> >>>>>>>> >>>>>>>> -Matthias >>>>>>>> >>>>>>>> >>>>>>>> On 8/22/18 10:36 PM, Dhruvil Shah wrote: >>>>>>>>> *"suppress" is the opposite of "allow", so >>>>>>>>> setting suppress.auto.create.topics=false would mean that we do >>>>>> _not_ >>>>>>>> allow >>>>>>>>> auto topic creation; when set to true, the server configuration >>>> will >>>>>>>>> determine whether we allow automatic creation or not.* >>>>>>>>> >>>>>>>>> Sorry, I meant suppress.auto.create.topics=true above to >> disallow >>>>>> auto >>>>>>>>> topic creation. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah < >>>> dhru...@confluent.io >>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> To be clear, we will allow auto topic creation only when >> server >>>>>> config >>>>>>>>>> auto.create.topics.enable=true and consumer config >>>>>>>>>> allow.auto.create.topics=true; when either is false, we would >>>> not >>>>>>> create >>>>>>>>>> the topic if it does not exist. >>>>>>>>>> >>>>>>>>>> "suppress" is the opposite of "allow", so >>>>>>>>>> setting suppress.auto.create.topics=false would mean that we >> do >>>>>> _not_ >>>>>>>> allow >>>>>>>>>> auto topic creation; when set to true, the server >> configuration >>>>>> will >>>>>>>>>> determine whether we allow automatic creation or not. >>>>>>>>>> >>>>>>>>>> I think "allow" is easier to understand but I am open to >>>>>> suggestions. >>>>>>>>>> >>>>>>>>>> - Dhruvil >>>>>>>>>> >>>>>>>>>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner < >>>>>>>>>> brandon.kirch...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> “allow=false” seems a bit more intuitive to me than >>>>>> “suppress=false” >>>>>>>>>>> >>>>>>>>>>> Brandon >>>>>>>>>>> >>>>>>>>>>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhih...@gmail.com> >>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> We may also consider : >>>>>>>>>>>> >>>>>>>>>>>> "suppress.auto.topic.creation" >>>>>>>>>>>> >>>>>>>>>>>> or >>>>>>>>>>>> >>>>>>>>>>>> "allow.auto.topic.creation" >>>>>>>>>>>> >>>>>>>>>>>> w.r.t. suppress or allow, I don't have strong opinion >> either. >>>>>> It's >>>>>>>> just >>>>>>>>>>> a >>>>>>>>>>>> matter of choosing the proper default value. >>>>>>>>>>>> >>>>>>>>>>>> Cheers >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah < >>>>>> dhru...@confluent.io >>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Matthias, >>>>>>>>>>>>> >>>>>>>>>>>>> Do you mean something like "suppress.auto.create.topic"? I >> am >>>>>>> leaning >>>>>>>>>>> a bit >>>>>>>>>>>>> towards "allow.auto.create.topics" but I don't have a >> strong >>>>>>>> preference >>>>>>>>>>>>> either. Let's wait to hear if anyone else has an opinion on >>>>>> this. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Dhruvil >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax < >>>>>>>> matth...@confluent.io >>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the KIP Dhruvil! >>>>>>>>>>>>>> >>>>>>>>>>>>>> I agree with Jason's comment. An alternative might be to >> use >>>>>>>>>>> "suppress" >>>>>>>>>>>>>> what would revert the logic of "allow". Not sure which one >>>> is >>>>>> more >>>>>>>>>>>>>> intuitive and I am fine with both (no personal >> preference). >>>>>> Just >>>>>>>>>>> wanted >>>>>>>>>>>>>> to mention it as an alternative. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Don't have any further comments/question so far. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote: >>>>>>>>>>>>>>> Hey Dhruvil, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would suggest using the verb "allow" rather than >> "enable. >>>>>> The >>>>>>>>>>>>> consumer >>>>>>>>>>>>>>> cannot enable auto topic creation because it is >> configured >>>> on >>>>>> the >>>>>>>>>>>>> broker. >>>>>>>>>>>>>>> All it can do is prevent it from happening if it is >>>> enabled. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Jason >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah < >>>>>>>> dhru...@confluent.io> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would like to start discussion on KIP-361 that >> proposes >>>> we >>>>>>> add a >>>>>>>>>>>>>> consumer >>>>>>>>>>>>>>>> configuration to disable auto topic creation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Link to the KIP: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+ >>>>>>>>>>>>>>>> Configuration+to+Disable+Auto+Topic+Creation >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Suggestions and feedback are welcome! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Dhruvil >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
