Just make sure they are not in the same consumer group by creating a unique value for group.id for each independent consumer.
/** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 31, 2016 at 9:42 AM, Patrick Viet <patrick.v...@gmail.com> wrote: > Hi, > > I've done some searching and can't find much on this topic. > Here is what I'm looking at: > > So a usual usage of kafka is to push messages in a topic, and then have > one/several consumers consume it (one per partition), in a "unique" way: a > message is consumed only once, and if it's consumed multiple times it's a > mistake, which we try to avoid. > > What if I'd like several consumers to get the same messages? > Here is a practical example: > > - Some app produces user action messages > - One entity (say, marketing) would like to consume them to do analysis > - Another entity (say, BI) would like to consume them too, to do some other > analysis > > So effectively both would get the full (duplicate) messages > > It seems that in the current high level consumer model, the messages only > go to one entity at a time, commits to zookeeper where it's at every X > messages read from partition Y, etc. My understanding is that if I want to > be able to read the messages in any other way, I'd have to make my own > consumer based on raw info from kafka and zookeeper. > > But maybe I'm wrong and it's totally possible in the current model to have > several consumers subscribe independently on a same topic, and consume then > with their own progression markers. If yes, a pointer to any documentation > / examples / discussions would be greatly appreciated > > -- > Best regards, > Patrick Viet >
