Hi Patrick, Can you use separate consumer groups to accomplish what you're looking for?
Scroll to the "consumers" section on this document for a more detailed description: https://kafka.apache.org/documentation.html#introduction On Mon, Oct 31, 2016 at 11:42 AM, Patrick Viet <patrick.v...@gmail.com> wrote: > Hi, > > I've done some searching and can't find much on this topic. > Here is what I'm looking at: > > So a usual usage of kafka is to push messages in a topic, and then have > one/several consumers consume it (one per partition), in a "unique" way: a > message is consumed only once, and if it's consumed multiple times it's a > mistake, which we try to avoid. > > What if I'd like several consumers to get the same messages? > Here is a practical example: > > - Some app produces user action messages > - One entity (say, marketing) would like to consume them to do analysis > - Another entity (say, BI) would like to consume them too, to do some other > analysis > > So effectively both would get the full (duplicate) messages > > It seems that in the current high level consumer model, the messages only > go to one entity at a time, commits to zookeeper where it's at every X > messages read from partition Y, etc. My understanding is that if I want to > be able to read the messages in any other way, I'd have to make my own > consumer based on raw info from kafka and zookeeper. > > But maybe I'm wrong and it's totally possible in the current model to have > several consumers subscribe independently on a same topic, and consume then > with their own progression markers. If yes, a pointer to any documentation > / examples / discussions would be greatly appreciated > > -- > Best regards, > Patrick Viet > -- *Samuel Taylor* Data Science *Square Root, Inc. <http://square-root.com/>* <http://square-root.com/>Square-Root.com <http://square-root.com/>
