Hi Nick, This looks like a valid use case and shouldn't fail. The only workaround I see is to create some dummy topics then.
On Fri, Jun 11, 2021 at 12:11 AM Martin, Nick J [US] (SP) < nick.mar...@ngc.com> wrote: > I’m trying to use the topic discovery feature of the Kafka Connector. The > problem I’m having is that Kafka Consumers fail to start if there are no > topics matching the topic regex when they start up. Is this intended > behavior? Is there some other property I could set to just continue > discovery until they find a matching topic? > > > > Background: > > My application uses dynamically generated topic names where specific > messages are sent on different topics based on some metadata in the > messages. A producer service reads the metadata, determines the topic the > data should be sent to, applies some validation logic to the topic name, > creates that topic if it doesn’t already exist, and then sends the data. > The problem is, when starting the whole stack with a fresh Kafka cluster, > my Flink job with the Kafka consumer can’t start until the producer service > has been started and has processed some data so that at least one matching > topic exists. This sort of startup order dependency is obviously > undesirable in a distributed microservice architecture. > > > > Are there existing features/configuration settings that solve this > problem? Should I open a ticket? >
