I'm trying to use the topic discovery feature of the Kafka Connector. The problem I'm having is that Kafka Consumers fail to start if there are no topics matching the topic regex when they start up. Is this intended behavior? Is there some other property I could set to just continue discovery until they find a matching topic?
Background: My application uses dynamically generated topic names where specific messages are sent on different topics based on some metadata in the messages. A producer service reads the metadata, determines the topic the data should be sent to, applies some validation logic to the topic name, creates that topic if it doesn't already exist, and then sends the data. The problem is, when starting the whole stack with a fresh Kafka cluster, my Flink job with the Kafka consumer can't start until the producer service has been started and has processed some data so that at least one matching topic exists. This sort of startup order dependency is obviously undesirable in a distributed microservice architecture. Are there existing features/configuration settings that solve this problem? Should I open a ticket?
