Do you have only one partition in the topic? The way Kafka works is that all
messages are first distributed into partitions in the topic and then the
consumers are distributed among them and they read them sequentially.
If you have only one partition in the topic, all your messages will be in it
but only one consumer can read messages per partition. If you have multiple
partitions, you can have that many consumers reading from it in parallel.
Let me know if that wasn't clear.
_____________________________
From: Kaushil Rambhia/ MUM/CORP/ ENGINEERING <kaushi...@pepperfry.com>
Sent: Wednesday, October 19, 2016 8:40 PM
Subject: Need to add & remove consumers dynamically in diffrent group and
consume consecutively.
To: <users@kafka.apache.org>
Hi guys,
i am using apache kafka with phprd kafka, i want to know how can i use
multiple Kafka consumers on same partition from different groups to consume
message parallel, say if consumer are c1,c2,c3 consuming single partition
0,
than if c1 is consuming from 0 offset than c2 should start from 1 and c3
from 2 and if any new consumer comes up it should start from latest i.e 4th
offset which is yet to be consumed by any consumer.
So in short all consumers should consume consecutively from kafka partition
and should not consumed same message again no matter any new consumer is
added or removed from different group.It should automatically select
consecutive offsets.
It would be good if anyone can recommend any solution or can this
be achieved by using kafka with some other tools like spark or something
else.
--
Regards,
Kaushil Rambhia
--
DISCLAIMER: The contents of this message may be legally privileged and
confidential and are for the use of the intended recipient(s) only. It
should not be read, copied and used by anyone other than the intended
recipient(s). If you have received this message in error, please
immediately notify the sender, preserve its confidentiality and delete it.
Before opening any attachments please check them for viruses and defects.
