I'm trying to use the topic discovery feature of the Kafka Connector. The 
problem I'm having is that Kafka Consumers fail to start if there are no topics 
matching the topic regex when they start up. Is this intended behavior? Is 
there some other property I could set to just continue discovery until they 
find a matching topic?

Background:
My application uses dynamically generated topic names where specific messages 
are sent on different topics based on some metadata in the messages. A producer 
service reads the metadata, determines the topic the data should be sent to, 
applies some validation logic to the topic name, creates that topic if it 
doesn't already exist, and then sends the data. The problem is, when starting 
the whole stack with a fresh Kafka cluster, my Flink job with the Kafka 
consumer can't start until the producer service has been started and has 
processed some data so that at least one matching topic exists. This sort of 
startup order dependency is obviously undesirable in a distributed microservice 
architecture.

Are there existing features/configuration settings that solve this problem? 
Should I open a ticket?

Reply via email to