Hi all, Tl;dr: I want to set up a subscription based on patterns that will match non-existent topics and receive all messages that match (e.g. test.*), missing first message to any new matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very low metadata.max.age.ms
I've been playing around with Kafka trying to build a PoC for a cluster event notification system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing some little nugget in the docs, or mis-understanding something. I've also searched around, but perhaps am using the wrong terms. I'd like to set up a consumer with a topic pattern along the lines of "test.*" then have it pick up all messages sent to topics that match that pattern. So far I'm missing the first new message sent to a new pattern. e.g. it will miss the first message sent to test.a, but catch subsequent ones. I've set the following: auto.create.topics.enable=true allow.auto.create.topics=true And in the consumer properties I've set metadata.max.age.ms very low (around 50). (I'm not sure what kind of performance problems doing this could do in production with multiple consumers.) Yet I'll still (almost always) miss the first message to a new topic in the pattern. If anyone has any ideas, or even better things to google for to help solve this I'd be most appreciative even if the result is "you just won't be able to do what you want" ;-) Many thanks in advance, Chris
