Hello Petr, Thank you for your reply.
I am not sure how your proposed solution could help my use case. Sending messages with the same key to the same partition will ensure that key : (message_type, value) that share a key will be consumed by the same consumer instance. However, my main concern was to separate messages based on message_type - hence I originally had one topic per type. Anyway, I settled on the idea to have one global topic containing messages of all types and avoid the hassle. Regards, /David On 20 August 2015 at 14:04, Petr Novak <oss.mli...@gmail.com> wrote: > Hi, > to my knowledge this is generally solved by partitioner which logs messages > with the same key to the same topic. It is not clear why you can't use this > approach. > > With Regards, > Petr > > On Thu, Aug 20, 2015 at 12:35 PM, David KOCH <dk...@ezakus.com> wrote: > > > Hello, > > > > Is it possible to consume messages from multiple topics in a way that > > messages which were written at the same time in different topics are also > > consumed at around the same time? > > > > Let me explain this with our use case. > > > > Our application logs keyed messages of N different types into N topics: > > > > - The same key may occur for 1 to N of the message types. > > - All messages with the same key are typically generated no more than > 10 > > seconds apart. > > - Some message types occur much less frequently than others. > > > > The job of the application consuming the topics is to re-concile messages > > from all topics based on the value of the key. It currently does this > using > > a single consumer which whitelists all of the N topics and uses a > > time-based eviction cache to collect messages which share a key. > > > > This works well for real time consumption, however, what happens when all > > topics are consumed from the beginning? A topic with very few messages > will > > be consumed faster than one with many messages making it unrealistic to > use > > the time-based eviction cache with a feasible timeout. Hence my initial > > question. > > > > I was thinking of using multiple consumers and somehow co-ordinating > these > > in order to make them advance in a way that messages (keys) which were > > written at around the same time are also consumed at around the same > time. > > > > Is there a straightforward way to accomplish something like this or is it > > too weird of a use case for Kafka as a whole? > > > > Alternatively, I could send everything into the same topic but > conceptually > > this is less elegant. > > > > Thank you, > > > > /David > > >
