Currently the KIP's scope is only topics that don't yet exist, and we have to cognizant of race conditions between tasks with the same connector. I think it is worthwhile to consider whether the KIP's scope should expand to also address *existing* partitions, though it may not be appropriate to have as much control when changing the topic settings for an existing topic. For example, changing the number of partitions (which the KIP considers a "topic-specific setting" even though technically it is not) shouldn't be done blindly due to the partitioning impacts, and IIRC you can't reduce them (which we could verify before applying). Also, I don't think the AdminClient currently allows changing the replication factory. I think changing the topic configs is less problematic both from what makes sense for connectors to verify/change and from what the AdminClient supports.
Even if we decide that it's not appropriate to change the settings on an existing topic, I do think it's advantageous to at least notify the connector (or task) prior to the first record sent to a given topic so that the connector can fail or issue a warning if it doesn't meet its requirements. Best regards, Randall On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi Randall, > > Thanks for the KIP. I like it > What happens when the target topic is already created but the configs do > not match? > i.e. wrong RF, num partitions, or missing / additional configs? Will you > attempt to apply the necessary changes or throw an error? > > Thanks! > Stephane > > > On 24/5/17, 5:59 am, "Mathieu Fenniak" <mathieu.fenn...@replicon.com> > wrote: > > Ah, yes, I see you a highlighted part that should've made this clear > to me the first read. :-) Much clearer now! > > By the way, enjoyed your Debezium talk in NYC. > > Looking forward to this Kafka Connect change; it will allow me to > remove a post-deployment tool that I hacked together for the purpose > of ensuring auto-created topics have the right config. > > Mathieu > > > On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> > wrote: > > Thanks for the quick feedback, Mathieu. Yes, the first configuration > rule > > whose regex matches will be applied, and no other rules will be > used. I've > > updated the KIP to try to make this more clear, but let me know if > it's > > still not clear. > > > > Best regards, > > > > Randall > > > > On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > > mathieu.fenn...@replicon.com> wrote: > > > >> Hi Randall, > >> > >> Awesome, very much looking forward to this. > >> > >> It isn't 100% clear from the KIP how multiple config-based rules > would > >> be applied; it looks like the first configuration rule whose regex > >> matches the topic name will be used, and no other rules will be > >> applied. Is that correct? (I wasn't sure if it might cascade > >> together multiple matching rules...) > >> > >> Looks great, > >> > >> Mathieu > >> > >> > >> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> > wrote: > >> > Hi, all. > >> > > >> > We recently added the ability for Kafka Connect to create > *internal* > >> topics > >> > using the new AdminClient, but it still would be great if Kafka > Connect > >> > could do this for new topics that result from source connector > records. > >> > I've outlined an approach to do this in "KIP-158 Kafka Connect > should > >> allow > >> > source connectors to set topic-specific settings for new topics". > >> > > >> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> set+topic-specific+settings+for+new+topics > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> set+topic-specific+settings+for+new+topics>* > >> > > >> > Please take a look and provide feedback. Thanks! > >> > > >> > Best regards, > >> > > >> > Randall > >> > > > >
