Re: KafkaProducer may send duplicated message sometimes

2017-03-31 Thread radai
removing the timeout completely will cause clients to just "hang" if the broker is unavailable (even worse, if only one broker out of a cluster goes down a producer that could have otherwise sent data to other brokers would fill up it memory with stuff waiting to be sent out to the one thats down).

Re: KafkaProducer may send duplicated message sometimes

2017-03-31 Thread Yang Cui
Hi Radai, Thanks for your reply, sincerely, I am so glad for that. In my opinion, I am clear that Kafka only provides the “at least once” semantics now, but I think Kafka should try it best to decrease the duplicated message case if it can. In this case described in my JIRA, I think Kafka c

Re: KafkaProducer may send duplicated message sometimes

2017-03-31 Thread radai
kafka (at least out of the box as it is now) is not an exactly-once system. its an at-least-once system, meaning the scenario you described (and similar ones involving socket disconnections, for example) exist by design. there is a KIP for adding exactly once guarantees (among other things) that y