Hi Justine, Thanks for the clarifications. I understand that auto-creation of topics will happen through CreateTopic request instead of metadata request. What I meant in earlier mail is producer client should not override broker config about auto-creation of topics. I agree with Harsha on other mail about this behavior. If auto-creation is disabled on broker, producer clients will never be allowed to create topics even if 'allow.auto.create.topics' is true in producer client.
On Wed, Aug 7, 2019 at 1:28 AM Justine Olshan <jols...@confluent.io> wrote: > > Hi Satish, > > Thanks for looking at the KIP. > > Yes, the producer will wait for the topic to be created before it can send > any messages to it. > > I would like to clarify "overriding" broker behavior. If the client enables > client-side autocreation, the only difference will be that the topic > auto-creation will no longer occur in the metadata request and will instead > come from a CreateTopic request on the producer. > Partitions and replication factor will be determined by the broker configs. > > Is this similar to what you were thinking? Please let me know if there is > something you think I missed. > > Thank you, > Justine > > On Tue, Aug 6, 2019 at 12:01 PM Satish Duggana <satish.dugg...@gmail.com> > wrote: > > > Hi Justine, > > Thanks for the KIP. This is a nice addition to the producer client > > without running admin-client’s create topic APIs. Does producer wait > > for the topic to be created successfully before it tries to publish > > messages to that topic? I assume that this will not throw an error > > that the topic does not exist. > > > > As mentioned by others, overriding broker behavior by producer looks > > to be a concern. IMHO, broker should have a way to use either default > > constraints or configure custom constraints before these can be > > overridden by clients but not vice versa. There should be an option on > > brokers whether those constraints can be overridden by producers or > > not. > > > > Thanks, > > Satish. > > > > On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan <jols...@confluent.io> > > wrote: > > > > > > Hi Harsha, > > > > > > After taking this all into consideration, I've updated the KIP to no > > longer > > > allow client-side configuration of replication factor and partitions. > > > Instead, the broker defaults will be used as long as the broker supports > > > KIP 464. > > > If the broker does not support this KIP, then the client can not create > > > topics on its own. (Behavior that exists now) > > > > > > I think this will help with your concerns. Please let me know if you > > > further feedback. > > > > > > Thank you, > > > Justine > > > > > > On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani <ka...@harsha.io> > > wrote: > > > > > > > Hi, > > > > Even with policies one needs to implement that, so for every user > > who > > > > doesn't want a producer to create topics or have limits around > > partitions & > > > > replication factor they have to implement a policy. > > > > The KIP is changing the behavior , it might not be introducing > > the > > > > new functionality but it will enable producers to override the create > > topic > > > > config settings on the broker. What I am asking for to provide a config > > > > that will disable auto creation of topics and if its enabled set some > > sane > > > > defaults so that clients can create a topic with in those limits. I > > don't > > > > see how this not related to this KIP. > > > > If the server config options as I suggested doesn't interest you > > at > > > > least have a default CreateTopicPolices in place. > > > > To give an example, In our environment we disable the > > > > auto.create.topic.enable and force users to go through a centralized > > > > service as we want capture more details about the topic creation and > > > > requirements. With this KIP, a producer can create a topic with no > > bounds. > > > > Another example max.message.size we define that at cluster level and > > one > > > > can override max.messsage.size at topic level, any misconfiguration at > > this > > > > will cause service degradation. Its not always about the rogue > > clients, > > > > users can easily misconfigure and can cause an outage. > > > > Again we can talk about CreateTopicPolicy but without having a default > > > > implementation and asking everyone to implement their own while > > changing > > > > the behavior in producer doesn't make sense to me. > > > > > > > > Thanks, > > > > Harsha > > > > > > > > > > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <ism...@juma.me.uk> > > wrote: > > > > > > > > > Hi Harsha, > > > > > > > > > > I mentioned policies and the authorizer. For example, with > > > > > CreateTopicPolicy, you can implement the limits you describe. If you > > have > > > > > ideas of how that should be improved, please submit a KIP. My point > > is > > > > that > > > > > this KIP is not introducing any new functionality with regards to > > what > > > > > rogue clients can do. It's using the existing protocol that is > > already > > > > > exposed via the AdminClient. So, I don't think we need to address it > > in > > > > > this KIP. Does that make sense? > > > > > > > > > > Ismael > > > > > > > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani <ka...@harsha.io> > > > > > wrote: > > > > > > > > > > Ismael, > > > > > Sure AdminClient can do that and we should've shipped a config or > > use the > > > > > existing one to block that. Not all users are yet to upgrade to > > > > AdminClient > > > > > and start using that to cause issues yet. In shared environment we > > should > > > > > allow server to set sane defaults and not allow every client to go > > ahead > > > > > create random no.of topic/partitions and replication factor. Even if > > the > > > > > users want to allow topic creation proposed in the KIP , it makes > > sense > > > > to > > > > > have some guards against the no.of partitions and replication factor. > > > > > Authorizer is not always an answer to block requests and having > > users set > > > > > server side configs to protect a multi-tenant environment is > > required. > > > > In a > > > > > non-secure environment Authorizer is a blunt instrument either you > > end up > > > > > blocking everyone or allowing everyone. > > > > > I am asking to have server side that allow clients to create topics > > or > > > > not > > > > > , if they are allowed set a ceiling on max no.of partitions and > > > > > replication-factor. > > > > > > > > > > -Harsha > > > > > > > > > > On Mon, Aug 5 2019 at 8:58 PM, <isma...@gmail.com> wrote: > > > > > > > > > > Harsha, > > > > > > > > > > Rogue clients can use the admin client to create topics and > > partitions. > > > > > ACLs and policies can help in that case as well as this one. > > > > > > > > > > Ismael > > > > > > > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani <ka...@harsha.io> > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > Thanks for the KIP. > > > > > "When server-side auto-creation is disabled, client-side > > auto-creation > > > > > will try to use client-side configurations" > > > > > If I understand correctly, this KIP is removing any server-side > > blocking > > > > > client auto creation of topic? > > > > > if so this will present potential issue of rogue client creating ton > > of > > > > > topic-partitions and potentially bringing down the service for > > everyone > > > > > > > > > > or > > > > > > > > > > degrade the service itself. > > > > > By reading the KIP its not clear to me that there is a clear way to > > block > > > > > auto creation topics of all together from clients by server side > > config. > > > > > Server side configs of default topic, partitions should take higher > > > > > precedence and client shouldn't be able to create a topic with higher > > > > > > > > > > no.of > > > > > > > > > > partitions, replication than what server config specifies. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan < > > jols...@confluent.io> > > > > > wrote: > > > > > > > > > > Hi all, > > > > > I made some changes to the KIP. Hopefully this configuration change > > > > > > > > > > will > > > > > > > > > > make things a little clearer. > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > > > > > > > > > Please let me know if you have any feedback or questions! > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe <cmcc...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > Hi Mickael, > > > > > > > > > > I think you bring up a good point. It would be better if we didn't > > ever > > > > > have to set up client-side configuration for this feature, and > > KIP-464 > > > > > would let us skip this entirely. > > > > > > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > > > > > > > > > Hi Mickael, > > > > > I agree that KIP-464 works on newer brokers, but I was a bit worried > > > > > > > > > > how > > > > > > > > > > things would play out on older brokers that* do not *have KIP 464 > > > > > > > > > > included. > > > > > > > > > > Is it enough to throw an error in this case when producer configs are > > > > > > > > > > not > > > > > > > > > > specified? > > > > > > > > > > I think the right thing to do would be to log an error message in the > > > > > client. We will need to have that capability in any case, to cover > > > > > scenarios like the client trying to auto-create a topic that they > > don't > > > > > have permission to create. Or a client trying to create a topic on a > > > > > > > > > > broker > > > > > > > > > > so old that CreateTopicsRequest is not supported. > > > > > > > > > > The big downside to relying on KIP-464 is that it is a very recent > > > > > > > > > > feature > > > > > > > > > > -- so recent that it hasn't even made its way to any official Apache > > > > > release. It's scheduled for the upcoming 2.4 release in a few months. > > > > > > > > > > So if you view this KIP as a step towards removing broker-side > > > > > auto-create, you might want to support older brokers just to > > accelerate > > > > > adoption, and hasten the day when we can finally flip broker-side > > > > > auto-create to off (or even remove it entirely). > > > > > > > > > > I have to agree, though, that having client-side configurations for > > > > > > > > > > number > > > > > > > > > > of partitions and replication factor is messy. Maybe it would be > > worth > > > > > > > > > > it > > > > > > > > > > to restrict support to post-KIP-464 brokers, if we could avoid > > creating > > > > > more configs. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison < > > > > > > > > > > mickael.mai...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > We can rely on KIP-464 which allows to omit the partition count or > > > > > replication factor when creating a topic. In that case, the broker > > > > > > > > > > defaults > > > > > > > > > > are used. > > > > > > > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan <jols...@confluent.io > > > > > > > > wrote: > > > > > > > > > > Michael, > > > > > That makes sense to me! > > > > > To clarify, in the current state of the KIP, the producer does not > > > > > > > > > > rely > > > > > > > > > > on > > > > > > > > > > the broker to autocreate--if the broker's config is disabled, then > > > > > > > > > > the > > > > > > > > > > producer can autocreate on its own with a create topic request (the > > > > > > > > > > same > > > > > > > > > > type of request the admin client uses). > > > > > However, if both configs are enabled, the broker will autocreate > > > > > > > > > > through > > > > > > > > > > a > > > > > > > > > > metadata request before the producer gets a chance. Of course, the > > way > > > > > > > > > > to > > > > > > > > > > avoid this, is to do as you suggested, and set > > > > > > > > > > the > > > > > > > > > > "allow_auto_topic_creation" field to false. > > > > > > > > > > I think the only thing we need to be careful with in this setup is > > > > > > > > > > without > > > > > > > > > > KIP 464, we can not use broker defaults for this topic. A user needs > > > > > > > > > > to > > > > > > > > > > specify the number of partition and replication factor in the config. > > > > > > > > > > An > > > > > > > > > > alternative to this is to have coded defaults for when these > > > > > > > > > > configs > > > > > > > > > > are > > > > > > > > > > unspecified, but it is not immediately apparent what these defaults > > > > > > > > > > should > > > > > > > > > > be. > > > > > > > > > > Thanks again for reading my KIP, > > > > > Justine > > > > > > > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison < > > > > > > > > > > mickael.mai...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the response! > > > > > In my opinion, it would be better if the producer did not rely at > > > > > > > > > > all > > > > > > > > > > on the broker auto create feature as this is what we're aiming to > > > > > deprecate. When requesting metadata we can set the > > > > > "allow_auto_topic_creation" field to false to avoid the broker auto > > > > > creation. Then if the topic is not existing, send a > > CreateTopicRequest. > > > > > > > > > > What do you think? > > > > > > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan < > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Currently the way it is implemented, the broker auto-creation > > > > > > > > > > configuration > > > > > > > > > > takes precedence. The producer will not use the CreateTopics > > > > > > > > > > request. > > > > > > > > > > (Technically it can--but the topic will already be created > > > > > > > > > > through > > > > > > > > > > the > > > > > > > > > > broker, so it will never try to create the topic.) It is possible to > > > > > change this however, and I'd be happy to > > > > > > > > > > discuss > > > > > > > > > > the > > > > > > > > > > benefits of this alternative. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison < > > > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > In case auto creation is enabled on both the client and server, > > > > > > > > > > will > > > > > > > > > > the producer still use the AdminClient (CreateTopics request) > > > > > > > > > > to > > > > > > > > > > create topics? and not rely on the broker auto create. I'm guessing > > the > > > > > answer is yes but can you make it explicit. > > > > > > > > > > Thank you > > > > > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan < > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Hi, > > > > > Just a friendly reminder to take a look at this KIP if you > > > > > > > > > > have > > > > > > > > > > the > > > > > > > > > > time. > > > > > > > > > > I was thinking about broker vs. client default precedence, > > > > > > > > > > and I > > > > > > > > > > think it > > > > > > > > > > makes sense to keep the broker as the default used when both > > > > > > > > > > client-side > > > > > > > > > > and broker-side defaults are configured. The idea is that > > > > > > > > > > there > > > > > > > > > > would be > > > > > > > > > > pretty clear documentation, and that many systems with > > > > > > > > > > configurations > > > > > > > > > > that > > > > > > > > > > the client could not change would likely have the auto-create > > > > > > > > > > default > > > > > > > > > > off. > > > > > > > > > > (In cloud for example). > > > > > > > > > > It also seems like in most cases, the consumer config > > > > > 'allow.auto.create.topics' was created to actually prevent > > > > > > > > > > the > > > > > > > > > > creation > > > > > > > > > > of > > > > > > > > > > topics, so the loss of creation functionality will not be a > > > > > > > > > > big > > > > > > > > > > problem. > > > > > > > > > > I'm happy to discuss any other compatibility problems or > > > > > > > > > > components > > > > > > > > > > of > > > > > > > > > > this KIP. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan < > > > > > > > > > > jols...@confluent.io > > > > > > > > > > wrote: > > > > > > > > > > Hello all, > > > > > > > > > > I was looking at this KIP again, and there is a decision I > > > > > > > > > > made > > > > > > > > > > that I > > > > > > > > > > think is worth discussing. > > > > > > > > > > In the case where both the broker and producer's > > > > > 'auto.create.topics.enable' are set to true, we have to > > > > > > > > > > choose > > > > > > > > > > either > > > > > > > > > > the > > > > > > > > > > broker configs or the producer configs for the replication > > > > > factor/partitions. > > > > > > > > > > Currently, the decision is to have the broker defaults take > > > > > > > > > > precedence. > > > > > > > > > > (It is easier to do this in the implementation.) It also > > > > > > > > > > makes > > > > > > > > > > some > > > > > > > > > > sense > > > > > > > > > > for this behavior to take precedence since this behavior > > > > > > > > > > already > > > > > > > > > > occurs as > > > > > > > > > > the default. > > > > > > > > > > However, I was wondering if it would be odd for those who > > > > > > > > > > can > > > > > > > > > > only > > > > > > > > > > see > > > > > > > > > > the > > > > > > > > > > producer side to set configs for replication > > > > > > > > > > factor/partitions > > > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > different results. Currently the documentation for the > > > > > > > > > > config > > > > > > > > > > states > > > > > > > > > > that > > > > > > > > > > the config values are only used when the broker config is > > > > > > > > > > not > > > > > > > > > > enabled, > > > > > > > > > > but > > > > > > > > > > this might not always be clear to the user. Changing the > > > > > > > > > > code > > > > > > > > > > to > > > > > > > > > > have > > > > > > > > > > the > > > > > > > > > > producer's configurations take precedence is possible, but > > > > > > > > > > I > > > > > > > > > > was > > > > > > > > > > wondering > > > > > > > > > > what everyone thought. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan < > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Just a quick update-- > > > > > > > > > > It seems that enabling both the broker and producer > > > > > > > > > > configs > > > > > > > > > > works > > > > > > > > > > fine, > > > > > > > > > > except that the broker configurations for partitions, > > > > > > > > > > replication > > > > > > > > > > factor > > > > > > > > > > take precedence. > > > > > I don't know if that is something we would want to > > > > > > > > > > change, but > > > > > > > > > > I'll be > > > > > > > > > > updating the KIP for now to reflect this. Perhaps we would > > > > > > > > > > want to > > > > > > > > > > add more > > > > > > > > > > to the documentation of the the producer configs to > > > > > > > > > > clarify. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan < > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Hi Colin, > > > > > > > > > > Thanks for looking at the KIP. I can definitely add to > > > > > > > > > > the > > > > > > > > > > title > > > > > > > > > > to > > > > > > > > > > make > > > > > > > > > > it more clear. > > > > > > > > > > It makes sense that both configurations could be turned > > > > > > > > > > on > > > > > > > > > > since > > > > > > > > > > there > > > > > > > > > > are many cases where the user can not control the > > > > > > > > > > server-side > > > > > > > > > > configurations. I was a little concerned about how both > > > > > > > > > > interacting > > > > > > > > > > would > > > > > > > > > > work out -- if there would be to many requests for new > > > > > > > > > > topics, > > > > > > > > > > for > > > > > > > > > > example. > > > > > > > > > > But it since it does make sense to allow both > > > > > > > > > > configurations > > > > > > > > > > enabled, I > > > > > > > > > > will test out some scenarios and I'll change the KIP to > > > > > > > > > > support > > > > > > > > > > this. > > > > > > > > > > I also agree with documentation about distinguishing the > > > > > > > > > > differences. I > > > > > > > > > > was having some trouble with the wording but I like the > > > > > > > > > > phrases > > > > > > > > > > "server-side" and "client-side." That's a good > > > > > > > > > > distinction I > > > > > > > > > > can > > > > > > > > > > use > > > > > > > > > > when > > > > > > > > > > describing. > > > > > > > > > > I'll try to update the KIP soon keeping everyone's input > > > > > > > > > > in > > > > > > > > > > mind. > > > > > > > > > > Thanks, > > > > > Justine > > > > > > > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe < > > > > > > > > > > cmcc...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP. This seems like a good step towards > > > > > > > > > > removing > > > > > > > > > > server-side topic auto-creation. > > > > > > > > > > We should add included "client-side" to the title of > > > > > > > > > > the KIP > > > > > > > > > > somewhere, > > > > > > > > > > to make it clear that we're talking about client-side > > > > > > > > > > auto > > > > > > > > > > creation. > > > > > > > > > > The KIP says: > > > > > > > > > > In order to automatically create topics with the > > > > > > > > > > producer, the > > > > > > > > > > producer's > > > > > > > > > > auto.create.topics.enable config must be set to true > > > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > config should be set to false > > > > > > > > > > From a user's point of view, this seems > > > > > > > > > > counter-intuitive. > > > > > > > > > > In > > > > > > > > > > order to > > > > > > > > > > auto-create topics the broker's > > > > > > > > > > auto.create.topics.enable > > > > > > > > > > config > > > > > > > > > > should be > > > > > > > > > > set to false? It seems like the server-side > > > > > > > > > > auto-create is > > > > > > > > > > unrelated to > > > > > > > > > > the client-side auto-create. We could have both turned > > > > > > > > > > on > > > > > > > > > > (and > > > > > > > > > > I'm > > > > > > > > > > sure > > > > > > > > > > that in the real world, people will try this > > > > > > > > > > configuration...) > > > > > > > > > > There's no > > > > > > > > > > reason not to support this, I think. > > > > > > > > > > We should add some documentation explaining the > > > > > > > > > > difference > > > > > > > > > > between > > > > > > > > > > server-side and client-side auto-creation. Without > > > > > > > > > > documentation, > > > > > > > > > > an admin > > > > > > > > > > might think that they had disabled all forms of > > > > > > > > > > auto-creation by > > > > > > > > > > setting > > > > > > > > > > the -side setting to false-- but this is not the case, > > > > > > > > > > of > > > > > > > > > > course. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > > > > > > > Hi Dhruvil, > > > > > > > > > > Thanks for reading the KIP! > > > > > That was the general idea for deprecation. We would > > > > > > > > > > log a > > > > > > > > > > warning > > > > > > > > > > when the > > > > > > > > > > config is enabled on the broker. > > > > > I also believe that there would be a change to > > > > > > > > > > documentation. > > > > > > > > > > If there is anything else that should be done, please > > > > > > > > > > let > > > > > > > > > > me > > > > > > > > > > know! > > > > > > > > > > Justine > > > > > > > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah < > > > > > > > > > > dhru...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP, this is great! > > > > > > > > > > Could you add some more information about what > > > > > > > > > > deprecating > > > > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > configuration means? Would we log a warning in the > > > > > > > > > > logs > > > > > > > > > > when > > > > > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > creation is enabled on the broker, for example? > > > > > > > > > > Thanks, > > > > > Dhruvil > > > > > > > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > wrote: > > > > > > > > > > Hello all, > > > > > > > > > > I'd like to start a discussion thread for KIP-487. This KIP plans to > > > > > deprecate the current system of > > > > > > > > > > auto-creating > > > > > > > > > > topics > > > > > > > > > > through requests to the metadata and give the > > > > > > > > > > producer the > > > > > > > > > > ability to > > > > > > > > > > automatically create topics instead. > > > > > > > > > > More information can be found here: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > > > > > > > Thank you, > > > > > Justine Olshan > > > > > > > > > > > > > > > >
