Hi Mickael, I agree that KIP-464 works on newer brokers, but I was a bit worried how things would play out on older brokers that* do not *have KIP 464 included. Is it enough to throw an error in this case when producer configs are not specified?
Thanks, Justine On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Justine, > > We can rely on KIP-464 which allows to omit the partition count or > replication factor when creating a topic. In that case, the broker > defaults are used. > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jols...@confluent.io> > wrote: > > > > Michael, > > That makes sense to me! > > To clarify, in the current state of the KIP, the producer does not rely > on > > the broker to autocreate--if the broker's config is disabled, then the > > producer can autocreate on its own with a create topic request (the same > > type of request the admin client uses). > > However, if both configs are enabled, the broker will autocreate through > a > > metadata request before the producer gets a chance. > > Of course, the way to avoid this, is to do as you suggested, and set the > > "allow_auto_topic_creation" field to false. > > > > I think the only thing we need to be careful with in this setup is > without > > KIP 464, we can not use broker defaults for this topic. A user needs to > > specify the number of partition and replication factor in the config. > > An alternative to this is to have coded defaults for when these configs > are > > unspecified, but it is not immediately apparent what these defaults > should > > be. > > > > Thanks again for reading my KIP, > > Justine > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <mickael.mai...@gmail.com > > > > wrote: > > > > > Hi Justine, > > > > > > Thanks for the response! > > > In my opinion, it would be better if the producer did not rely at all > > > on the broker auto create feature as this is what we're aiming to > > > deprecate. When requesting metadata we can set the > > > "allow_auto_topic_creation" field to false to avoid the broker auto > > > creation. Then if the topic is not existing, send a > > > CreateTopicRequest. > > > > > > What do you think? > > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <jols...@confluent.io> > > > wrote: > > > > > > > > Currently the way it is implemented, the broker auto-creation > > > configuration > > > > takes precedence. The producer will not use the CreateTopics request. > > > > (Technically it can--but the topic will already be created through > the > > > > broker, so it will never try to create the topic.) > > > > It is possible to change this however, and I'd be happy to discuss > the > > > > benefits of this alternative. > > > > > > > > Thank you, > > > > Justine > > > > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison < > > > mickael.mai...@gmail.com> > > > > wrote: > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > In case auto creation is enabled on both the client and server, > will > > > > > the producer still use the AdminClient (CreateTopics request) to > > > > > create topics? and not rely on the broker auto create. > > > > > I'm guessing the answer is yes but can you make it explicit. > > > > > > > > > > Thank you > > > > > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan < > jols...@confluent.io> > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > Just a friendly reminder to take a look at this KIP if you have > the > > > time. > > > > > > > > > > > > I was thinking about broker vs. client default precedence, and I > > > think it > > > > > > makes sense to keep the broker as the default used when both > > > client-side > > > > > > and broker-side defaults are configured. The idea is that there > > > would be > > > > > > pretty clear documentation, and that many systems with > configurations > > > > > that > > > > > > the client could not change would likely have the auto-create > default > > > > > off. > > > > > > (In cloud for example). > > > > > > > > > > > > It also seems like in most cases, the consumer config > > > > > > 'allow.auto.create.topics' was created to actually prevent the > > > creation > > > > > of > > > > > > topics, so the loss of creation functionality will not be a big > > > problem. > > > > > > > > > > > > I'm happy to discuss any other compatibility problems or > components > > > of > > > > > > this KIP. > > > > > > > > > > > > Thank you, > > > > > > Justine > > > > > > > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan < > jols...@confluent.io > > > > > > > > > wrote: > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > I was looking at this KIP again, and there is a decision I made > > > that I > > > > > > > think is worth discussing. > > > > > > > > > > > > > > In the case where both the broker and producer's > > > > > > > 'auto.create.topics.enable' are set to true, we have to choose > > > either > > > > > the > > > > > > > broker configs or the producer configs for the replication > > > > > > > factor/partitions. > > > > > > > > > > > > > > Currently, the decision is to have the broker defaults take > > > precedence. > > > > > > > (It is easier to do this in the implementation.) It also makes > some > > > > > sense > > > > > > > for this behavior to take precedence since this behavior > already > > > > > occurs as > > > > > > > the default. > > > > > > > > > > > > > > However, I was wondering if it would be odd for those who can > only > > > see > > > > > the > > > > > > > producer side to set configs for replication factor/partitions > and > > > see > > > > > > > different results. Currently the documentation for the config > > > states > > > > > that > > > > > > > the config values are only used when the broker config is not > > > enabled, > > > > > but > > > > > > > this might not always be clear to the user. Changing the code > to > > > have > > > > > the > > > > > > > producer's configurations take precedence is possible, but I > was > > > > > wondering > > > > > > > what everyone thought. > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan < > > > jols...@confluent.io> > > > > > > > wrote: > > > > > > > > > > > > > >> Just a quick update-- > > > > > > >> > > > > > > >> It seems that enabling both the broker and producer configs > works > > > > > fine, > > > > > > >> except that the broker configurations for partitions, > replication > > > > > factor > > > > > > >> take precedence. > > > > > > >> I don't know if that is something we would want to change, but > > > I'll be > > > > > > >> updating the KIP for now to reflect this. Perhaps we would > want to > > > > > add more > > > > > > >> to the documentation of the the producer configs to clarify. > > > > > > >> > > > > > > >> Thank you, > > > > > > >> Justine > > > > > > >> > > > > > > >> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan < > > > jols...@confluent.io> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Hi Colin, > > > > > > >>> > > > > > > >>> Thanks for looking at the KIP. I can definitely add to the > title > > > to > > > > > make > > > > > > >>> it more clear. > > > > > > >>> > > > > > > >>> It makes sense that both configurations could be turned on > since > > > > > there > > > > > > >>> are many cases where the user can not control the server-side > > > > > > >>> configurations. I was a little concerned about how both > > > interacting > > > > > would > > > > > > >>> work out -- if there would be to many requests for new > topics, > > > for > > > > > example. > > > > > > >>> But it since it does make sense to allow both configurations > > > > > enabled, I > > > > > > >>> will test out some scenarios and I'll change the KIP to > support > > > this. > > > > > > >>> > > > > > > >>> I also agree with documentation about distinguishing the > > > > > differences. I > > > > > > >>> was having some trouble with the wording but I like the > phrases > > > > > > >>> "server-side" and "client-side." That's a good distinction I > can > > > use > > > > > when > > > > > > >>> describing. > > > > > > >>> > > > > > > >>> I'll try to update the KIP soon keeping everyone's input in > mind. > > > > > > >>> > > > > > > >>> Thanks, > > > > > > >>> Justine > > > > > > >>> > > > > > > >>> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe < > cmcc...@apache.org > > > > > > > > > wrote: > > > > > > >>> > > > > > > >>>> Hi Justine, > > > > > > >>>> > > > > > > >>>> Thanks for the KIP. This seems like a good step towards > > > removing > > > > > > >>>> server-side topic auto-creation. > > > > > > >>>> > > > > > > >>>> We should add included "client-side" to the title of the KIP > > > > > somewhere, > > > > > > >>>> to make it clear that we're talking about client-side auto > > > creation. > > > > > > >>>> > > > > > > >>>> The KIP says: > > > > > > >>>> > In order to automatically create topics with the > producer, the > > > > > > >>>> producer's > > > > > > >>>> > auto.create.topics.enable config must be set to true and > the > > > > > broker > > > > > > >>>> config should be set to false > > > > > > >>>> > > > > > > >>>> From a user's point of view, this seems counter-intuitive. > In > > > > > order to > > > > > > >>>> auto-create topics the broker's auto.create.topics.enable > config > > > > > should be > > > > > > >>>> set to false? It seems like the server-side auto-create is > > > > > unrelated to > > > > > > >>>> the client-side auto-create. We could have both turned on > (and > > > I'm > > > > > sure > > > > > > >>>> that in the real world, people will try this > configuration...) > > > > > There's no > > > > > > >>>> reason not to support this, I think. > > > > > > >>>> > > > > > > >>>> We should add some documentation explaining the difference > > > between > > > > > > >>>> server-side and client-side auto-creation. Without > > > documentation, > > > > > an admin > > > > > > >>>> might think that they had disabled all forms of > auto-creation by > > > > > setting > > > > > > >>>> the -side setting to false-- but this is not the case, of > > > course. > > > > > > >>>> > > > > > > >>>> best, > > > > > > >>>> Colin > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > > > >>>> > Hi Dhruvil, > > > > > > >>>> > > > > > > > >>>> > Thanks for reading the KIP! > > > > > > >>>> > That was the general idea for deprecation. We would log a > > > warning > > > > > > >>>> when the > > > > > > >>>> > config is enabled on the broker. > > > > > > >>>> > I also believe that there would be a change to > documentation. > > > > > > >>>> > If there is anything else that should be done, please let > me > > > know! > > > > > > >>>> > > > > > > > >>>> > Justine > > > > > > >>>> > > > > > > > >>>> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah < > > > > > dhru...@confluent.io> > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > > >>>> > > Hi Justine, > > > > > > >>>> > > > > > > > > >>>> > > Thanks for the KIP, this is great! > > > > > > >>>> > > > > > > > > >>>> > > Could you add some more information about what > deprecating > > > the > > > > > > >>>> broker > > > > > > >>>> > > configuration means? Would we log a warning in the logs > when > > > > > auto > > > > > > >>>> topic > > > > > > >>>> > > creation is enabled on the broker, for example? > > > > > > >>>> > > > > > > > > >>>> > > Thanks, > > > > > > >>>> > > Dhruvil > > > > > > >>>> > > > > > > > > >>>> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < > > > > > > >>>> jols...@confluent.io> > > > > > > >>>> > > wrote: > > > > > > >>>> > > > > > > > > >>>> > > > Hello all, > > > > > > >>>> > > > > > > > > > >>>> > > > I'd like to start a discussion thread for KIP-487. > > > > > > >>>> > > > This KIP plans to deprecate the current system of > > > > > auto-creating > > > > > > >>>> topics > > > > > > >>>> > > > through requests to the metadata and give the > producer the > > > > > > >>>> ability to > > > > > > >>>> > > > automatically create topics instead. > > > > > > >>>> > > > > > > > > > >>>> > > > More information can be found here: > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > > > >>>> > > > > > > > > > >>>> > > > Thank you, > > > > > > >>>> > > > Justine Olshan > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>> > > > > > > > > >
