Konstantine, thanks for the updates. I wonder if we should take your proposal one step further and make this pluggable. Your include/exclude regexes are great out-of-the-box features, but it may be valuable to plug-in more sophisticated logic to handle topic creation.
Instead of enabling/disabling the feature as a whole, the default TopicCreator (or whatever) could be a nop. Then we include a RegexTopicCreator with your proposed behavior. This would be almost indistinguishable from your current KIP from a user's perspective, but would enable plug-in TopicCreators that do some of the things you have listed in the Rejected Alternatives, e.g. to automatically adjust the replication factor based on the number of nodes, etc. My team leverages Connect's plug-ins in other places to enable seamless integration with the rest of our platform. We would definitely use a topic creation hook if one existed. In particular, we have a concept of "topic profiles" that we could use here. Ryanne On Thu, Dec 12, 2019 at 2:00 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > I've taken a second look to KIP-158 after syncing with Randall Hauch, who > was the original author of the proposal, and I have updated the KIP in > place. > > The main new features of this updated KIP-158 is the introduction of groups > of configs that can be composed and the ability to match topics to these > groups via the use of regex. The design builds on top of the existing > definition of config groups used in single message transformations (SMT) > and therefore I'm hoping that the approach fits well in Kafka Connect's > current configuration capabilities. > > The new proposal aims to strike a good balance between requiring to > explicitly set the configs for each possible topic or having a > one-size-fits-all default set of properties for all the topics a connector > may create during runtime. > > > The updated KIP-158 can be found under the same page as the old one: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics > > I've intentionally changed the title here in this thread to avoid confusion > with the threads that discussed KIP-158 previously. > Looking forward to your comments and hoping we can pick up this work from > the very good starting point that was reached in the previous discussions. > > > Best, > Konstantine >
