On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote: > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > wrote: > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > >> > >> Hi Colin, > >> "Hmm... I'm not sure I follow. Users don't have to build their own > >> tooling, right? They can use any of the shell scripts that we've shipped > >> in the last few releases. For example, if any of your users run it, this > >> shell script will delete all of the topics from your non-security-enabled > >> cluster: > >> > >> > >> > >> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server > >> localhost:9092 --list 2>/dev/null > >> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > >> --bootstrap-server > >> localhost:9092 --delete > >> --topic > >> > >> > >> > >> They will need to fill in the correct bootstrap servers list, of course, > >> not localhost. This deletion script will work on some pretty old brokers, > >> even back to the 0.10 releases. It seems a little odd to trust your users > >> with this power, but not trust them to avoid changing a particular > >> configuration key." > >> > >> > >> > >> > >> The above will blocked by the server if we set delete.topic.enable to > >> false and thats exactly what I am asking for. > >> > >> > >> > > > > > > > > Hi Harsha, > > > > > > > > > > I was wondering if someone was going to bring up that configuration :) > > > > > > > > > > it's an interesting complication, but globally disabling topic deletion is > > not very practical for most use-cases. > > > > > > > > > > In any case, there are plenty of other bad things that users with full > > permissions can do that aren't blocked by any server configuration. For > > example, they can delete every record in every topic. I can write a script > > for that too, and there's no server configuration you can set to disable > > it. Or I could simply create hundreds of thousands of topics, until > > cluster performance becomes unacceptable (this will be even more of a > > problem if someone configured delete.topic.enable as false). Or publish > > bad data to every topic, etc. etc. > > > > > > > > > > The point I'm trying to make here is that you can't rely on these kind of > > server-side configurations for security. At most, they're a way to set up > > certain very simple policies. But the policies are so simple that they're > > hardly ever useful any more. > > > > > > > > > > For example, if the problem you want to solve is that you want a user to > > only be able to create 50 topics and not delete anyone else's topics, you > > can solve that with a CreateTopicsPolicy that limits the number of topics, > > and some ACLs. There's no combination of auto.create.topics.enable and > > delete.topic.enable that will help here. > > > > > > > > > > Hi Colin, > > Well you gave the example that a user can delete the topics > just by running that script :). > > I understand there are open APIs in Kafka and can lead to rogue clients > taking advantage of it without proper security in place. > > What I am asking so far in this thread is , this KIP is changing the > producer behavior and its not backward compatible.
The change is backwards compatible. The default will still be server-side topic auto-creation, just like now. You will have to specifically change the producer config to get the new behavior. > We can still achieve > the main goal of the KIP which is to change MetadataRequest creating > topics and send a CreateTopicRequest from Producer and also keep the > server side config to have precedence. This KIP originally written to > have server side preference and there is not much explanation why it > changed to have Producer side preference. > > Arguing that AdminClient can do that and so we are going to make > Producer do the same doesn't make sense. > > > > > > > > > > > > > > > > >> > >> > >> "The downside is that if we wanted to check a server side configuration > >> before sending the create topics request, the code would be more complex. > >> The behavior would also not be consistent with how topic auto-creation is > >> handled in Kafka Streams." > >> > >> > >> > >> > >> I am not sure why you need to check server side configuration before > >> sending create topics request. A user enables producer side config to > >> create topics. > >> Producer sends a request to the broker and if the broker has > >> auto.topic.create.enable to true (default) it will allow creation of > >> topics. If it set to false it returns error back to the client. > >> > >> > > > > > > > > auto.topic.create.enable has never affected CreateTopicsRequest. If you > > submit a CreateTopicsRequest and you are authorized, a topic will be > > created, regardless of what the value of auto.topic.create.enable is. This > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?) > > > > > > > >> > >> > >> I don't see how this behavior will be different in Kafka streams. By > >> default server allows the topic creation and with this KIP, It will only > >> allow creation of topic when both producer and server side are turned on. > >> Its exactly the same behavior in KIP-361. > >> > >> > >> > > > > > > > > I suggest running a streams job as a test. It creates the topics it needs > > using AdminClient. The value of auto.topic.create.enable is not relevant. > > Whether it is true or false, the topics will still be created. > > > > > > > >> > >> > >> "In general, it would be nice if we could keep the security and access > >> control model simple and not introduce a lot of special cases and > >> exceptions. Kafka has basically converged on using ACLs and > >> CreateTopicPolicy classes to control who can create topics and where. > >> Adding more knobs seems like a step backwards, especially when the > >> proposed knobs don't work consistently across components, and don't > >> provide true security." This is not about access control at all. Shipping > >> sane defaults should be prioritized. > >> > >> > >> > > > > > > > > I don't think the default is really the question here. I think everyone > > agrees that the default for client-side auto-topic creation should be off. > > > > > > > > > > > > My point goes back to KIP-361 why we kept the server side config to > have preference. We can keep bringing back the discussion of security > policies but a producer client overiding server side setting is not > just security policy issue . > > "Producer client can override server side setting and not consumer and > also producer can only override creation of topic but no the > replication and partitions. " This doesn't make sense to me and why we > want to introduce such a confusing behavior. > > > > > > > > > The scenarios that you're describing (such as dealing with a poorly > > configured client that tries to create topics it should not) do seem to be > > about access control. > > > > > > > >> > >> > >> We keep talking about CreateTopicPolicy and yet we don't have default one > >> and asking every user of Kafka implement their own doesn't make sense > >> here. > >> > >> > >> > > > > > > > > The point of CreateTopicPolicy is exactly that you can implement your own, > > though. It's a way of customizing what the policy is in your specific > > cluster. > > > > > > > > > > I agree that most people don't want to write a custom policy. But most of > > those people are satisfied with just setting up ACLs to set up simple > > policies like this user can create topics, that user can't, etc. That's > > why I suggested adding support for ACLs to insecure clusters might be > > helpful. > > > > > > > > > > Also, just as a side note, wouldn't it would be impossible for us to > > specify a default CreateTopicsPolicy that did anything at this point > > anyway without breaking backwards compatibility? Maybe I'm > > misunderstanding the proposal. > > > > > > > >> > >> > >> I am asking about exact behavior that we shipped for consumer side https:/ > >> / cwiki. apache. org/ confluence/ display/ KAFKA/ > >> KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation > >> ( > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation > >> ) > >> > >> > >> > >> > >> I still didn't get any response why this behavior shouldn't be exactly > >> like Kafka consumer and why do we want to have producer to overrider > >> server side config and while not allowing consumer to do so. > >> We are not even allowing the same contract and this will cause more > >> confusion from the users standpoint. > >> > >> > > > > > > > > Hmm. The consumer already has a way to override the server side > > configuration. See KIP-361: Add Consumer Configuration to Disable Auto > > Topic Creation. This lets the consumer skip auto-creating topics, even if > > auto-creation is enabled on the broker. > > > > > > > > > > Yes that's what I am saying and it doesn't allow topic creation if the > server side auto-creation is disabled and in consumer its enabled. I > would like to see the exact behavior for producer as stated in KIP-361. > > > > > > > > > To be fair, the KIP should probably discuss why we don't want to implement > > client-side topic creation in the consumer in "rejected alternatives." > > Maybe Justine can add more context here in the KIP. > > > > > > > > > > The last time we talked about this, the reasoning was that we wanted to > > eventually get rid of consumer-side auto-topic creation entirely, and so > > it wasn't worth implementing an improved way of doing it. Producer side > > auto topic-creation, in contrast, will probably stick around, although > > we'd like to deprecate and remove the problematic server-side mechanism > > over time. > > > > > > > > > > We should implement the producer side topic creation with proper create > topic request and I am in favor of this KIP. > > All I am asking is don't make the producer to override server side > config and keep it similar to KIP-361 just like consumer, it honors > what set on server side which takes the precedence. Changing this > behavior in producer is backward incompatible and will catch users by > surprise. > > All arguments for enforcing producer side overriding config can still > be achieved. Server side auto creation turned on by default and any > new producer client setting their auto creation config to true will > create topics in default behavior and any user who set server side to > false and upgrades to latest kafka with this KIP will not see any > unwanted behavior from clients. IMO this is not a unreasonable request > on this KIP and this requested behavior is exactly what the KIP > initially proposed. > > Thanks, > > Harsha > > > > > > > > > best, > > Colin > > > > > >> > >> > >> On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org ( > >> cmcc...@apache.org ) > wrote: > >> > >> > >> > >>> > >>> > >>> On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote: > >>> > >>> > >>> > >>>> > >>>> > >>>> Not sure how the AdminClient applies here, It is an external API and is > >>>> not part of KafkaProducer so any user who updates to latest version of > >>>> Kafka with this feature get to create the topics. > >>>> They have to build a tooling around AdminClient allow themselves to > >>>> > >>>> > >>> > >>> > >>> > >>> create > >>> > >>> > >>> > >>>> > >>>> > >>>> topics. > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> Hi Harsha, > >>> > >>> > >>> > >>> > >>> Hmm... I'm not sure I follow. Users don't have to build their own tooling, > >>> right? They can use any of the shell scripts that we've shipped in the > >>> last few releases. For example, if any of your users run it, this shell > >>> script will delete all of the topics from your non-security-enabled > >>> cluster: > >>> > >>> > >>> > >>> > >>> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server > >>> localhost:9092 --list 2>/dev/null > >>> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > >>> --bootstrap-server > >>> localhost:9092 --delete > >>> --topic > >>> > >>> > >>> > >>> They will need to fill in the correct bootstrap servers list, of course, > >>> not localhost. This deletion script will work on some pretty old brokers, > >>> even back to the 0.10 releases. It seems a little odd to trust your users > >>> with this power, but not trust them to avoid changing a particular > >>> configuration key. > >>> > >>> > >>> > >>>> > >>>> > >>>> There is no behavior in Kafka producer that allowed users to delete the > >>>> topics or delete the records. So citing them as an example doesn't makes > >>>> sense in this context. > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> I think Kafka Streams is relevant here. After all, it's software that we > >>> ship as part of the official Kafka release. And it auto-creates topics > >>> even when auto.create.topics.enable is set to false on the broker. > >>> > >>> > >>> > >>> > >>> I think that auto.create.topics.enable was never intended as a security > >>> setting to control access. It was intended as a way to disable the > >>> broker-side auto-create feature specifically, because there were some > >>> problems with that specific feature. Broker-side auto-creation is > >>> frustrating because it's on by default, and because it can auto-create > >>> topics even if the producer or consumer didn't explicitly ask for that. > >>> Neither of those problems applies to this KIP: producers have to > >>> specifically opt in, and it won't be on by default. Basically, we think > >>> that client-side auto-creation is actually a lot better-- hence this KIP > >>> :) > >>> > >>> > >>> > >>>> > >>>> > >>>> But there is > >>>> a functionality which allowed creation of topics if they don't exist in > >>>> > >>>> > >>> > >>> > >>> > >>> the > >>> > >>> > >>> > >>>> > >>>> > >>>> cluster and this behavior could be controlled by a config on the server > >>>> side. Now with this KIP we are allowing producer to make that decision > >>>> without any gateway on the server via configs. Any user who is not aware > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> of > >>> > >>> > >>> > >>>> > >>>> > >>>> such changes > >>>> can accidentally create these topics and we are essentially removing a > >>>> config that exists in brokers today to block this accidental creation and > >>>> allowing clients to take control. > >>>> > >>>> > >>> > >>> > >>> > >>> Again, I hope I'm not misinterpreting, but I don't see how this can be > >>> turned on accidentally. The user would have to specifically turn this on > >>> in the producer by setting the configuration key. > >>> > >>> > >>> > >>>> > >>>> > >>>> I still didn't get any positives of not having server side configs? if > >>>> you > >>>> want to turn them on and allow any client to create topics set the > >>>> default > >>>> to true > >>>> and allow users who doesn't want to have this feature let them turn them > >>>> off. It will be the exact behavior as it is today , as far as producer is > >>>> concerned. I am not > >>>> understanding why not having server side configs to gateways such a hard > >>>> requirement and this behavior exists today. As far I am concerned this > >>>> breaks the backward compatibility. > >>>> > >>>> > >>> > >>> > >>> > >>> The downside is that if we wanted to check a server side configuration > >>> before sending the create topics request, the code would be more complex. > >>> The behavior would also not be consistent with how topic auto-creation is > >>> handled in Kafka Streams. > >>> > >>> > >>> > >>> > >>> In general, it would be nice if we could keep the security and access > >>> control model simple and not introduce a lot of special cases and > >>> exceptions. Kafka has basically converged on using ACLs and > >>> CreateTopicPolicy classes to control who can create topics and where. > >>> Adding more knobs seems like a step backwards, especially when the > >>> proposed knobs don't work consistently across components, and don't > >>> provide true security. > >>> > >>> > >>> > >>> > >>> Maybe we should support an insecure mode where there are still users and > >>> ACLs. That would help people who don't want to set up Kerberos, etc. but > >>> still want to protect against misconfigured clients or accidents. Hadoop > >>> has something like this, and I think it was useful. NFS also supported > >>> (supports?) a mode where you just pass whatever user ID you want and the > >>> system believes you. These things clearly don't protect against malicious > >>> users, but they can help set up policies when needed. > >>> > >>> > >>> > >>> best, > >>> Colin > >>> > >>> > >>>> > >>>> > >>>> Thanks, > >>>> Harsha > >>>> > >>>> > >>>> > >>>> On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org ( > >>>> cmcc...@apache.org ) > wrote: > >>>> > >>>> > >>>> > >>>>> > >>>>> > >>>>> Hi Harsha, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Thanks for taking a look. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> I'm not sure I follow the discussion about AdminClient. > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> KafkaAdminClient > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> has been around for about 2 years at this point as a public class. > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> There > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> are many programs that use it to automatically create topics. Kafka > >>>>> Streams does this, for example. If any of your users run Kafka > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> Streams, > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> they will be auto-creating topics, regardless of what setting you use > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> for > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> auto.create.topics.enable. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> A big part of the annoyance of auto-topic creation right now is that > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> it is > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> on by default. The new configuration proposed by KIP-487 wouldn't be. > >>>>> Users would have to explicitly opt in to the new behavior of > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> client-side > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> topic creation. If you run without security, you're already putting a > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> huge > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> amount of trust in your users. For example, you trust them not to > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> delete > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> records with the kafka-delete-records. sh ( > >>>>> http://kafka-delete-records.sh/ > >>>>> ) command, or delete topics with kafka-topics. sh ( > >>>>> http://kafka-topics.sh/ > >>>>> ). Trusting them not to set a certain config value seems minor in > >>>>> comparison, right? > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> best, > >>>>> Colin > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote: > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> Hi, > >>>>>> Even with policies one needs to implement that, so for every > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> user who > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> doesn't want a producer to create topics or have limits around > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> partitions & > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> replication factor they have to implement a policy. The KIP is changing > >>>>>> the behavior , it might not be introducing > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> the > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> new functionality but it will enable producers to override the create > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> topic > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> config settings on the broker. What I am asking for to provide a > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> config > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> that will disable auto creation of topics and if its enabled set some > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> sane > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> defaults so that clients can create a topic with in those limits. I > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> don't > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> see how this not related to this KIP. > >>>>>> If the server config options as I suggested doesn't interest > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> you at > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> least have a default CreateTopicPolices in place. > >>>>>> To give an example, In our environment we disable the > >>>>>> auto.create.topic.enable and force users to go through a centralized > >>>>>> service as we want capture more details about the topic creation and > >>>>>> requirements. With this KIP, a producer can create a topic with no > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> bounds. > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> Another example max.message.size we define that at cluster level > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> and one > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> can override max.messsage.size at topic level, any misconfiguration > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> at > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> this > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> will cause service degradation. Its not always about the rogue > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> clients, > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> users can easily misconfigure and can cause an outage. Again we can > >>>>>> talk > >>>>>> about CreateTopicPolicy but without having a > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> default > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> implementation and asking everyone to implement their own while > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> changing > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> the behavior in producer doesn't make sense to me. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> Harsha > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. uk ( > >>>>>> ism...@juma.me.uk ) > > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> wrote: > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Harsha, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I mentioned policies and the authorizer. For example, with > >>>>>>> CreateTopicPolicy, you can implement the limits you describe. If > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> you > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> have > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ideas of how that should be improved, please submit a KIP. My > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> point is > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> that > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> this KIP is not introducing any new functionality with regards to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> what > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> rogue clients can do. It's using the existing protocol that is > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> already > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> exposed via the AdminClient. So, I don't think we need to address > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> it in > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> this KIP. Does that make sense? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Ismael > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> kafka@ harsha. io ( ka...@harsha.io ) > > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Ismael, > >>>>>>> Sure AdminClient can do that and we should've shipped a config or > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> use > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> the > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> existing one to block that. Not all users are yet to upgrade to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> AdminClient > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> and start using that to cause issues yet. In shared environment we > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> should > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> allow server to set sane defaults and not allow every client to go > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> ahead > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> create random no.of topic/partitions and replication factor. Even > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> if > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> the > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> users want to allow topic creation proposed in the KIP , it makes > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> sense to > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> have some guards against the no.of partitions and replication > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> factor. > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Authorizer is not always an answer to block requests and having > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> users > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> set > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> server side configs to protect a multi-tenant environment is > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> required. > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> In a > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> non-secure environment Authorizer is a blunt instrument either you > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> end > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> up > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> blocking everyone or allowing everyone. > >>>>>>> I am asking to have server side that allow clients to create > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> topics or > >>> > >>> > >>> > >>>> > >>>>> > >>>>> > >>>>> not > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> , if they are allowed set a ceiling on max no.of partitions and > >>>>>>> replication-factor. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -Harsha > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( > >>>>>>> isma...@gmail.com ) > >>>>>>> > wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Harsha, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Rogue clients can use the admin client to create topics and > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> partitions. > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ACLs and policies can help in that case as well as this one. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Ismael > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. io > >>>>>>> ( > >>>>>>> ka...@harsha.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> Thanks for the KIP. > >>>>>>> "When server-side auto-creation is disabled, client-side > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> auto-creation > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> will try to use client-side configurations" > >>>>>>> If I understand correctly, this KIP is removing any server-side > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> blocking > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> client auto creation of topic? > >>>>>>> if so this will present potential issue of rogue client creating > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> ton of > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> topic-partitions and potentially bringing down the service for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> everyone > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> or > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> degrade the service itself. > >>>>>>> By reading the KIP its not clear to me that there is a clear way to > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> block > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto creation topics of all together from clients by server side > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> config. > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Server side configs of default topic, partitions should take higher > >>>>>>> precedence and client shouldn't be able to create a topic with > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> higher > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> no.of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> partitions, replication than what server config specifies. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Harsha > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi all, > >>>>>>> I made some changes to the KIP. Hopefully this configuration change > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> will > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> make things a little clearer. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) > >>>>>>> KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Please let me know if you have any feedback or questions! > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. org ( > >>>>>>> cmcc...@apache.org ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Mickael, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I think you bring up a good point. It would be better if we didn't > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> ever > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> have to set up client-side configuration for this feature, and > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> KIP-464 > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> would let us skip this entirely. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Mickael, > >>>>>>> I agree that KIP-464 works on newer brokers, but I was a bit > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> worried > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> how > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> things would play out on older brokers that* do not *have KIP 464 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> included. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Is it enough to throw an error in this case when producer configs > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> are > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> not > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> specified? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I think the right thing to do would be to log an error message in > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> the > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> client. We will need to have that capability in any case, to cover > >>>>>>> scenarios like the client trying to auto-create a topic that they > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> don't > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> have permission to create. Or a client trying to create a topic on > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> a > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> broker > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> so old that CreateTopicsRequest is not supported. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> The big downside to relying on KIP-464 is that it is a very recent > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> feature > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- so recent that it hasn't even made its way to any official > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> Apache > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> release. It's scheduled for the upcoming 2.4 release in a few > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> months. > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> So if you view this KIP as a step towards removing broker-side > >>>>>>> auto-create, you might want to support older brokers just to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> accelerate > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> adoption, and hasten the day when we can finally flip broker-side > >>>>>>> auto-create to off (or even remove it entirely). > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I have to agree, though, that having client-side configurations for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> number > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> of partitions and replication factor is messy. Maybe it would be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> worth > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> it > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to restrict support to post-KIP-464 brokers, if we could avoid > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> creating > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> more configs. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> best, > >>>>>>> Colin > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> We can rely on KIP-464 which allows to omit the partition count or > >>>>>>> replication factor when creating a topic. In that case, the broker > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> defaults > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> are used. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Michael, > >>>>>>> That makes sense to me! > >>>>>>> To clarify, in the current state of the KIP, the producer does not > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> rely > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> on > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the broker to autocreate--if the broker's config is disabled, then > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer can autocreate on its own with a create topic request (the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> same > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> type of request the admin client uses). > >>>>>>> However, if both configs are enabled, the broker will autocreate > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> through > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> a > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> metadata request before the producer gets a chance. Of course, the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> way > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> avoid this, is to do as you suggested, and set > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> "allow_auto_topic_creation" field to false. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I think the only thing we need to be careful with in this setup is > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> without > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> KIP 464, we can not use broker defaults for this topic. A user > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> needs > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> specify the number of partition and replication factor in the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> config. > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> An > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> alternative to this is to have coded defaults for when these > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configs > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> are > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> unspecified, but it is not immediately apparent what these defaults > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> should > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> be. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks again for reading my KIP, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for the response! > >>>>>>> In my opinion, it would be better if the producer did not rely at > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> all > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> on the broker auto create feature as this is what we're aiming to > >>>>>>> deprecate. When requesting metadata we can set the > >>>>>>> "allow_auto_topic_creation" field to false to avoid the broker auto > >>>>>>> creation. Then if the topic is not existing, send a > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> CreateTopicRequest. > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> What do you think? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Currently the way it is implemented, the broker auto-creation > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configuration > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> takes precedence. The producer will not use the CreateTopics > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> request. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> (Technically it can--but the topic will already be created > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> through > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> broker, so it will never try to create the topic.) It is possible > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> to > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> change this however, and I'd be happy to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> discuss > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> benefits of this alternative. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for the KIP! > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> In case auto creation is enabled on both the client and server, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> will > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the producer still use the AdminClient (CreateTopics request) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> create topics? and not rely on the broker auto create. I'm > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> guessing the > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> answer is yes but can you make it explicit. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi, > >>>>>>> Just a friendly reminder to take a look at this KIP if you > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> have > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> time. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I was thinking about broker vs. client default precedence, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> and I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> think it > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> makes sense to keep the broker as the default used when both > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> client-side > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> and broker-side defaults are configured. The idea is that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> there > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> would be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> pretty clear documentation, and that many systems with > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configurations > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the client could not change would likely have the auto-create > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> default > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> off. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> (In cloud for example). > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> It also seems like in most cases, the consumer config > >>>>>>> ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' > >>>>>>> was > >>>>>>> created to actually prevent > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> creation > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> topics, so the loss of creation functionality will not be a > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> big > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> problem. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I'm happy to discuss any other compatibility problems or > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> components > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> this KIP. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hello all, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I was looking at this KIP again, and there is a decision I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> made > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> that I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> think is worth discussing. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> In the case where both the broker and producer's > >>>>>>> 'auto.create.topics.enable' are set to true, we have to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> choose > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> either > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> broker configs or the producer configs for the replication > >>>>>>> factor/partitions. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Currently, the decision is to have the broker defaults take > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> precedence. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> (It is easier to do this in the implementation.) It also > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> makes > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> some > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> sense > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> for this behavior to take precedence since this behavior > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> already > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> occurs as > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the default. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> However, I was wondering if it would be odd for those who > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> can > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> see > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer side to set configs for replication > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> factor/partitions > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> and > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> see > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> different results. Currently the documentation for the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> config > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> states > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the config values are only used when the broker config is > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> not > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> enabled, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> but > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> this might not always be clear to the user. Changing the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> code > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> have > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer's configurations take precedence is possible, but > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> was > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wondering > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> what everyone thought. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Just a quick update-- > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> It seems that enabling both the broker and producer > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configs > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> works > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> fine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> except that the broker configurations for partitions, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> replication > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> factor > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> take precedence. > >>>>>>> I don't know if that is something we would want to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> change, but > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I'll be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> updating the KIP for now to reflect this. Perhaps we would > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> want to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> add more > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to the documentation of the the producer configs to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> clarify. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Colin, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for looking at the KIP. I can definitely add to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> title > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> make > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> it more clear. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> It makes sense that both configurations could be turned > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> on > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> since > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> there > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> are many cases where the user can not control the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> server-side > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configurations. I was a little concerned about how both > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> interacting > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> would > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> work out -- if there would be to many requests for new > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> topics, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> example. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> But it since it does make sense to allow both > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configurations > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> enabled, I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> will test out some scenarios and I'll change the KIP to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> support > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> this. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I also agree with documentation about distinguishing the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> differences. I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> was having some trouble with the wording but I like the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> phrases > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> "server-side" and "client-side." That's a good > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> distinction I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> can > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> use > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> when > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> describing. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I'll try to update the KIP soon keeping everyone's input > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> in > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> mind. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> cmccabe@ apache. org ( cmcc...@apache.org ) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for the KIP. This seems like a good step towards > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> removing > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> server-side topic auto-creation. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> We should add included "client-side" to the title of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the KIP > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> somewhere, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> to make it clear that we're talking about client-side > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> creation. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> The KIP says: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> In order to automatically create topics with the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer, the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer's > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto.create.topics.enable config must be set to true > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> and > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> broker > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> config should be set to false > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> From a user's point of view, this seems > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> counter-intuitive. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> In > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> order to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto-create topics the broker's > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto.create.topics.enable > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> config > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> should be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> set to false? It seems like the server-side > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto-create is > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> unrelated to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the client-side auto-create. We could have both turned > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> on > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> (and > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I'm > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> sure > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> that in the real world, people will try this > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configuration...) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> There's no > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> reason not to support this, I think. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> We should add some documentation explaining the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> difference > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> between > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> server-side and client-side auto-creation. Without > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> documentation, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> an admin > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> might think that they had disabled all forms of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto-creation by > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> setting > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the -side setting to false-- but this is not the case, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> course. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> best, > >>>>>>> Colin > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Dhruvil, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for reading the KIP! > >>>>>>> That was the general idea for deprecation. We would > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> log a > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> warning > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> when the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> config is enabled on the broker. > >>>>>>> I also believe that there would be a change to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> documentation. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> If there is anything else that should be done, please > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> let > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> me > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> know! > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Justine > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> dhruvil@ confluent. io ( dhru...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi Justine, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks for the KIP, this is great! > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Could you add some more information about what > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> deprecating > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> broker > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> configuration means? Would we log a warning in the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> logs > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> when > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> topic > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> creation is enabled on the broker, for example? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Dhruvil > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan < > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> jolshan@ confluent. io ( jols...@confluent.io ) > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hello all, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> I'd like to start a discussion thread for KIP-487. This KIP plans > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> to > >>> > >>> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> deprecate the current system of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> auto-creating > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> topics > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> through requests to the metadata and give the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> producer the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ability to > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> automatically create topics instead. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> More information can be found here: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) > >>>>>>> KIP-487%3A+Automatic+Topic+Creation+on+Producer > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Justine Olshan > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > > > > > >
