Hi Chris Since you are only missing the very first message sent to that topic, I wonder its related to " auto.offset.reset" consumer setting. This setting controls what happens when no offset is present for a topic (most commonly the case when there has been no consumer yet for that topic)
From documentation: "Use auto.offset.reset to define the behaviour of the consumer when there is no committed position (which would be the case when the group is first initialized) or when an offset is out of range. You can choose either to reset the position to the "earliest" offset or the "latest" offset (the default). You can also select "none" if you would rather set the initial offset yourself and you are willing to handle out of range errors manually." With default behaviour of "latest" there is a possibility that consumer is not yet ready to listen to messages by the time 1st message was sent but then its ready and starts consuming from 2nd (latest since its stared listening). You can prove or disprove this theory of mine by changing default to "earliest" and see if you always get messages from beginning once default is overridden (or not). Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/ On 19/07/2019, 14:01, "Chris Hampson" <ch...@hampson.dev> wrote: Hi all, Tl;dr: I want to set up a subscription based on patterns that will match non-existent topics and receive all messages that match (e.g. test.*), missing first message to any new matching topic, set auto.create.topics.enable=true and allow.auto.create.topics=true and very low metadata.max.age.ms I've been playing around with Kafka trying to build a PoC for a cluster event notification system. I can't /quite/ get it doing what I'm after (at the moment). No doubt this is me missing some little nugget in the docs, or mis-understanding something. I've also searched around, but perhaps am using the wrong terms. I'd like to set up a consumer with a topic pattern along the lines of "test.*" then have it pick up all messages sent to topics that match that pattern. So far I'm missing the first new message sent to a new pattern. e.g. it will miss the first message sent to test.a, but catch subsequent ones. I've set the following: auto.create.topics.enable=true allow.auto.create.topics=true And in the consumer properties I've set metadata.max.age.ms very low (around 50). (I'm not sure what kind of performance problems doing this could do in production with multiple consumers.) Yet I'll still (almost always) miss the first message to a new topic in the pattern. If anyone has any ideas, or even better things to google for to help solve this I'd be most appreciative even if the result is "you just won't be able to do what you want" ;-) Many thanks in advance, Chris
