Hi, I finalized the implementation https://github.com/apache/kafka/pull/20384: if auto-creation of topics is disabled, an exception is thrown when a required topic does not exist. If the topic exists, there is an additional validation to ensure that it has the correct cleanup policy.
Please review the KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect and PR. I’ll start the voting if there are no concerns On Tue, Aug 26, 2025 at 4:58 PM Anton Liauchuk <anton93...@gmail.com> wrote: > Hello everyone > > I'd like to discuss a KIP > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect > > small POC with discussion: https://github.com/apache/kafka/pull/20384 > > Motivation: > > Kafka Connect requires several internal topics (config, offsets, and > status) for storing connector metadata and state. By default, these topics > are automatically created by the worker if they are missing. > However, in some scenarios this behavior can be problematic: > - internal topics may already contain information about a large number of > connectors. > - if one or more of the original topics are accidentally removed or > renamed, Kafka Connect will create them silently. > - this can lead to unexpected states — e.g., if the config topic exists > but the offsets topic was removed, all connectors may start from scratch. > We are going to introduce safer mode where Kafka Connect fails fast > instead of silently creating new topics, allowing to troubleshoot > misconfigurations and prevent accidental data loss or resets. > > Thank you! >
