I know we can have that thing but my concern is i don’t want to read every new consumer joining in to read from starting, i want them to read from where any last message consumed and that too consecutively by all consumers.
On Wed, Oct 19, 2016 at 9:34 PM, Sharninder Khera <sharnin...@gmail.com> wrote: > Do you have only one partition in the topic? The way Kafka works is that > all messages are first distributed into partitions in the topic and then > the consumers are distributed among them and they read them sequentially. > If you have only one partition in the topic, all your messages will be in > it but only one consumer can read messages per partition. If you have > multiple partitions, you can have that many consumers reading from it in > parallel. > Let me know if that wasn't clear. > > _____________________________ > From: Kaushil Rambhia/ MUM/CORP/ ENGINEERING <kaushi...@pepperfry.com> > Sent: Wednesday, October 19, 2016 8:40 PM > Subject: Need to add & remove consumers dynamically in diffrent group and > consume consecutively. > To: <users@kafka.apache.org> > > > Hi guys, > i am using apache kafka with phprd kafka, i want to know how can i use > multiple Kafka consumers on same partition from different groups to consume > message parallel, say if consumer are c1,c2,c3 consuming single partition > 0, > than if c1 is consuming from 0 offset than c2 should start from 1 and c3 > from 2 and if any new consumer comes up it should start from latest i.e 4th > offset which is yet to be consumed by any consumer. > > So in short all consumers should consume consecutively from kafka partition > and should not consumed same message again no matter any new consumer is > added or removed from different group.It should automatically select > consecutive offsets. > > It would be good if anyone can recommend any solution or can this > be achieved by using kafka with some other tools like spark or something > else. > > -- > Regards, > Kaushil Rambhia > > -- > > > > DISCLAIMER: The contents of this message may be legally privileged and > confidential and are for the use of the intended recipient(s) only. It > should not be read, copied and used by anyone other than the intended > recipient(s). If you have received this message in error, please > immediately notify the sender, preserve its confidentiality and delete it. > Before opening any attachments please check them for viruses and defects. > > > > > -- > > > > DISCLAIMER: The contents of this message may be legally privileged and > confidential and are for the use of the intended recipient(s) only. It > should not be read, copied and used by anyone other than the intended > recipient(s). If you have received this message in error, please > immediately notify the sender, preserve its confidentiality and delete it. > Before opening any attachments please check them for viruses and defects. > -- Regards, Kaushil Rambhia
