Re: Causes for Kafka messages being unexpectedly delivered more than once? The 'exactly once' semantic

2017-04-13 Thread Dmitry Goldenberg
ind, that exactly-once feature will not help > you with you scenario -- and also note, that there is no message brokers > out their, that could give stronger semantics than Kafka -- I hope that > he description about offset commit and fail over explain why it's not > possible to gi

Re: Causes for Kafka messages being unexpectedly delivered more than once? The 'exactly once' semantic

2017-04-13 Thread Dmitry Goldenberg
n > restart, if you want to receive the message again or not... Kafka cannot > know this. > > If you want to get the full details about exactly-once, you can have a > look into the KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transaction

Re: Causes for Kafka messages being unexpectedly delivered more than once? The 'exactly once' semantic

2017-04-13 Thread Dmitry Goldenberg
ion, well that > means > I've already received that message in the past. > > You could even do an insert of the topic+partition+offset of the > message > payload as the insert (instead of the full message) if you know for > sure > that your message

Causes for Kafka messages being unexpectedly delivered more than once? The 'exactly once' semantic

2017-04-13 Thread Dmitry Goldenberg
Hi all, I was wondering if someone could list some of the causes which may lead to Kafka delivering the same messages more than once. We've looked around and we see no errors to notice, yet intermittently, we see messages being delivered more than once. Kafka documentation talks about the below

Failing to write to Kafka (partitions "don't have a leader")

2015-09-28 Thread Dmitry Goldenberg
We're running into this issue in a clustered environment where we're trying to send messages to Kafka and are getting the below error. Can someone explain what might be causing it and what the error message means (Failed to send data since partitions [,8] don't have a leader) ? --