Hiya!

I've been trying to merge https://github.com/zendesk/maxwell/pull/457,
which adds a much-requested feature of Maxwell, that of being able to
have a topic-per-mysql-table.  When we receive a row we programmatically
generate the topic name, and the first thing we do is call
`KafkaProducer#partitionsFor(topic)`, so that we know how to partition
the data.

The problem I'm running into is in trying to detect the case where a
topic doesn't exist.  If auto-creation is on, `partitionsFor()` seems to
correctly auto-create the topic, but if auto-creation is off the
behavior is kinda wonky; kafka goes into a metadata-fetch loop, logging

"Error while fetching metadata with correlation id 573 
:{topic=UNKNOWN_TOPIC_OR_PARTITION}"

but then ultimately throwing me back a `TimeoutException` after 60 tries
or so.

I can rescue/rethrow the TimeoutException, but it seems like there might
be a better way that I'm missing.  Any ideas?  I'd ideally just like a
way to fail fast and clean when the topic doesn't exist (and
auto-creation is off).

Thanks,
Ben Osheroff
zendesk.com



Attachment: signature.asc
Description: PGP signature

Reply via email to