You could just catch the exception but if this is per row, that is probably prohibitively expensive.
Doesn't the binlog get "create table" events? Wouldn't that be a better time to create the topic? -- Andy On Mon, Oct 24, 2016 at 2:32 PM, Ben Osheroff <b...@zendesk.com.invalid> wrote: > Hiya! > > I've been trying to merge https://github.com/zendesk/maxwell/pull/457, > which adds a much-requested feature of Maxwell, that of being able to > have a topic-per-mysql-table. When we receive a row we programmatically > generate the topic name, and the first thing we do is call > `KafkaProducer#partitionsFor(topic)`, so that we know how to partition > the data. > > The problem I'm running into is in trying to detect the case where a > topic doesn't exist. If auto-creation is on, `partitionsFor()` seems to > correctly auto-create the topic, but if auto-creation is off the > behavior is kinda wonky; kafka goes into a metadata-fetch loop, logging > > "Error while fetching metadata with correlation id 573 > :{topic=UNKNOWN_TOPIC_OR_PARTITION}" > > but then ultimately throwing me back a `TimeoutException` after 60 tries > or so. > > I can rescue/rethrow the TimeoutException, but it seems like there might > be a better way that I'm missing. Any ideas? I'd ideally just like a > way to fail fast and clean when the topic doesn't exist (and > auto-creation is off). > > Thanks, > Ben Osheroff > zendesk.com > > > >
