Hi, Thanks for the feedback. AS1: Yes, the same worker.properties will be used for topic creation as for the worker itself. AS2: I will implement it. Thank you!
On Fri, Feb 27, 2026 at 5:15 PM Andrew Schofield <[email protected]> wrote: > Hi Anton, > Thanks for the KIP. Makes sense to me. Just two comments before I vote. > > AS1: KIP-1147 aligned the command-line properties of all of the > command-line tools. The Connect tools are a little different than all of > the other command-line tools, so I just want to make sure that I understand > the way the new tools is supposed to be used. Consistency is good. > > I think what you have intended for the new tool is: > bin/connect-internal-topics.sh create --worker-config > <worker_properties_file> > > Here are the existing ones: > bin/connect-standalone.sh <worker_properties_file> > <connector.properties......> > bin/connect-distributed.sh <worker_properties_file> > bin/connect-plugin-path.sh list --worker-config <worker_properties_file> > > Is it the case that the same worker.properties will be used for the topic > creation as for the worker itself? I think the answer is yes, but wanted to > check. > > AS2: Please make sure that you implement -h and --help for the new tool, > in a similar manner as bin/connect-plugin-path.sh. > > > Thanks, > Andrew > > On 2025/09/02 10:50:12 Anton Liauchuk wrote: > > Thanks for your question. > > > > The *auto.create.topics.enable* configuration will remain a broker-level > > setting. The new proposed configuration,* internal.topics.creation*, will > > be a Kafka Connect–level setting. > > > > For Kafka Connect internal topics, there is a requirement that these > topics > > must have *cleanup.policy=compact*. In my proposal, the new > > *internal.topics.creation* configuration takes precedence. Let’s review > > some examples: > > > > > > 1. > > > > *Internal topics do not exist, and internal.topics.creation=false > > (regardless of auto.create.topics.enable=true/false)* > > > > The Kafka Connect worker will not start, and an exception will be > thrown. > > 2. > > > > *Internal topics do not exist, and internal.topics.creation=true > > (regardless of auto.create.topics.enable=true/false)* > > > > Kafka Connect will attempt to create the required topics. If topic > > creation fails, the worker will not start - this is the current > behavior in > > Kafka Connect. > > 3. > > > > *Internal topics already exist, and > internal.topics.creation=true/false > > (regardless of auto.create.topics.enable=true/false)* > > > > No difference in behavior: Kafka Connect will validate the cleanup > > policy of these topics, and if valid, the worker will start. > > > > So as you can see, the broker-level configuration does not affect the > > behavior of the new configuration. > > > > On Tue, Sep 2, 2025 at 11:57 AM Kuan Po Tseng <[email protected]> > wrote: > > > > > Hi Anton, > > > > > > I have a quick question: how does this setting interact with > > > auto.create.topics.enable? > > > If I set internal.topics.creation=false while > auto.create.topics.enable is > > > true, > > > will the Connect internal topic still be created automatically? > > > > > > Best, > > > Kuan-Po Tseng > > > > > > On 2025/09/02 08:44:27 Anton Liauchuk wrote: > > > > Hi, > > > > > > > > I finalized the implementation > > > https://github.com/apache/kafka/pull/20384: > > > > if auto-creation of topics is disabled, an exception is thrown when a > > > > required topic does not exist. If the topic exists, there is an > > > additional > > > > validation to ensure that it has the correct cleanup policy. > > > > > > > > Please review the KIP > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect > > > > and PR. I’ll start the voting if there are no concerns > > > > > > > > On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <[email protected] > > > > > wrote: > > > > > > > > > Hello everyone > > > > > > > > > > I'd like to discuss a KIP > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect > > > > > > > > > > small POC with discussion: > https://github.com/apache/kafka/pull/20384 > > > > > > > > > > Motivation: > > > > > > > > > > Kafka Connect requires several internal topics (config, offsets, > and > > > > > status) for storing connector metadata and state. By default, these > > > topics > > > > > are automatically created by the worker if they are missing. > > > > > However, in some scenarios this behavior can be problematic: > > > > > - internal topics may already contain information about a large > number > > > of > > > > > connectors. > > > > > - if one or more of the original topics are accidentally removed or > > > > > renamed, Kafka Connect will create them silently. > > > > > - this can lead to unexpected states — e.g., if the config topic > exists > > > > > but the offsets topic was removed, all connectors may start from > > > scratch. > > > > > We are going to introduce safer mode where Kafka Connect fails fast > > > > > instead of silently creating new topics, allowing to troubleshoot > > > > > misconfigurations and prevent accidental data loss or resets. > > > > > > > > > > Thank you! > > > > > > > > > > > > > > >
