Hi, to my knowledge this is generally solved by partitioner which logs messages with the same key to the same topic. It is not clear why you can't use this approach.
With Regards, Petr On Thu, Aug 20, 2015 at 12:35 PM, David KOCH <dk...@ezakus.com> wrote: > Hello, > > Is it possible to consume messages from multiple topics in a way that > messages which were written at the same time in different topics are also > consumed at around the same time? > > Let me explain this with our use case. > > Our application logs keyed messages of N different types into N topics: > > - The same key may occur for 1 to N of the message types. > - All messages with the same key are typically generated no more than 10 > seconds apart. > - Some message types occur much less frequently than others. > > The job of the application consuming the topics is to re-concile messages > from all topics based on the value of the key. It currently does this using > a single consumer which whitelists all of the N topics and uses a > time-based eviction cache to collect messages which share a key. > > This works well for real time consumption, however, what happens when all > topics are consumed from the beginning? A topic with very few messages will > be consumed faster than one with many messages making it unrealistic to use > the time-based eviction cache with a feasible timeout. Hence my initial > question. > > I was thinking of using multiple consumers and somehow co-ordinating these > in order to make them advance in a way that messages (keys) which were > written at around the same time are also consumed at around the same time. > > Is there a straightforward way to accomplish something like this or is it > too weird of a use case for Kafka as a whole? > > Alternatively, I could send everything into the same topic but conceptually > this is less elegant. > > Thank you, > > /David >
