Thanks Alok, I'd come across that and begun playing with different values for that. Tweaking it has done exactly what I was after. I'm now getting the desired behaviour.
Thanks again for the speedy response, Cheers, Chris > On 22 Jul 2019, at 09:36, Alok Dwivedi <alok.dwiv...@instaclustr.com> wrote: > > Hi Chris > Since you are only missing the very first message sent to that topic, I > wonder its related to " auto.offset.reset" consumer setting. This setting > controls what happens when no offset is present for a topic (most commonly > the case when there has been no consumer yet for that topic) > > From documentation: > "Use auto.offset.reset to define the behaviour of the consumer when there is > no committed position (which would be the case when the group is first > initialized) or when an offset is out of range. You can choose either to > reset the position to the "earliest" offset or the "latest" offset (the > default). You can also select "none" if you would rather set the initial > offset yourself and you are willing to handle out of range errors manually." > > With default behaviour of "latest" there is a possibility that consumer is > not yet ready to listen to messages by the time 1st message was sent but then > its ready and starts consuming from 2nd (latest since its stared listening). > You can prove or disprove this theory of mine by changing default to > "earliest" and see if you always get messages from beginning once default is > overridden (or not). > > Thanks > Alok Dwivedi > Senior Consultant > https://www.instaclustr.com/platform/ > > > On 19/07/2019, 14:01, "Chris Hampson" <ch...@hampson.dev> wrote: > > Hi all, > > Tl;dr: I want to set up a subscription based on patterns that will match > non-existent topics and receive all messages that match (e.g. test.*), > missing first message to any new matching topic, set > auto.create.topics.enable=true and allow.auto.create.topics=true and very low > metadata.max.age.ms > > I've been playing around with Kafka trying to build a PoC for a cluster > event notification system. I can't /quite/ get it doing what I'm after (at > the moment). No doubt this is me missing some little nugget in the docs, or > mis-understanding something. I've also searched around, but perhaps am using > the wrong terms. > > I'd like to set up a consumer with a topic pattern along the lines of > "test.*" then have it pick up all messages sent to topics that match that > pattern. So far I'm missing the first new message sent to a new pattern. > > e.g. it will miss the first message sent to test.a, but catch subsequent > ones. > > I've set the following: > auto.create.topics.enable=true > allow.auto.create.topics=true > > And in the consumer properties I've set metadata.max.age.ms very low > (around 50). (I'm not sure what kind of performance problems doing this could > do in production with multiple consumers.) > > Yet I'll still (almost always) miss the first message to a new topic in > the pattern. > > If anyone has any ideas, or even better things to google for to help solve > this I'd be most appreciative even if the result is "you just won't be able > to do what you want" ;-) > > Many thanks in advance, > > Chris >
