Thanks Adam for a very detailed explanation.
Regards,
Kris
On Fri, Jun 19, 2015 at 7:23 AM, Adam Shannon
wrote:
> Basically it boils down to the fact that distributed computers and their
> networking are not reliable. [0] So, in order to ensure that messages do
> infact get across there are cas
Hi Shayne,
Each consumer has one partition to consume from and duplicates messages are
being received from different consumers.
Thanks,
Kris
On Fri, Jun 19, 2015 at 7:41 AM, Shayne S wrote:
> Duplicate messages might be due to network issues, but it is worthwhile to
> dig deeper.
>
> It sounds
Duplicate messages might be due to network issues, but it is worthwhile to
dig deeper.
It sounds like the problem happens when you have 3 partitions and 3
consumers. Based on my understanding (still learning), each consumer should
have it's own partition to consume. Can you verify this while your
Basically it boils down to the fact that distributed computers and their
networking are not reliable. [0] So, in order to ensure that messages do
infact get across there are cases where duplicates have to be sent.
Take for example this simple experiment, given three servers A, B, and C. A
sends a
Thanks Adam for your response.
I will have a mechanism to handle duplicates on the service consuming the
messages.
Just curious, if there is a way to identify the cause for receiving
duplicates.
I mean any log file that could help with this?
Regards,
Kris
On Wed, Jun 17, 2015 at 8:24 AM, Adam Sha
This is actually an expected consequence of using distributed systems. The
kafka FAQ has a good answer
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactly-oncemessagingfromKafka
?
On Tue, Jun 16, 2015 at 11:06 PM, Kris K wrote:
> Hi,
>
> While testing message delivery usi
Hi,
While testing message delivery using kafka, I realized that few duplicate
messages got delivered by the consumers in the same consumer group (two
consumers got the same message with few milli-seconds difference). However,
I do not see any redundancy at the producer or broker. One more observat
