removing the timeout completely will cause clients to just "hang" if the broker is unavailable (even worse, if only one broker out of a cluster goes down a producer that could have otherwise sent data to other brokers would fill up it memory with stuff waiting to be sent out to the one thats down).
if youre seeing this issue often, and your brokers are stable, then likely your timeout values are too short vs your latency/load. but like i said, as the code is right now this situation is unavoidable (just unlikely under good configurations and a stable environment) and client code needs to be written to account for this - either "dedup" things you read or make sure read side-effects are idempotent in some other way. longer-term when KIP-98 is implemented you could consider switching to it On Fri, Mar 31, 2017 at 9:28 AM, Yang Cui <y...@freewheel.tv> wrote: > Hi Radai, > Thanks for your reply, sincerely, I am so glad for that. > In my opinion, I am clear that Kafka only provides the “at least once” > semantics now, but I think Kafka should try it best to decrease the > duplicated message case if it can. In this case described in my JIRA, I > think Kafka can remove timeout check code to decrease the possibility of > duplicating message. > > > On 31/03/2017, 11:17 PM, "radai" <radai.rosenbl...@gmail.com> wrote: > > kafka (at least out of the box as it is now) is not an exactly-once > system. > its an at-least-once system, meaning the scenario you described (and > similar ones involving socket disconnections, for example) exist by > design. > > there is a KIP for adding exactly once guarantees (among other things) > that > you can read here - > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging > > > On Fri, Mar 31, 2017 at 4:07 AM, Yang Cui <y...@freewheel.tv> wrote: > > > Hi All, > > There is a JIRA issue which I submitted: > https://issues.apache.org/ > > jira/browse/KAFKA-4951; > > It is about the Sending duplicated message by KafkaProducer, > Could you > > please check it? > > Thanks. > > > > Yang Cui > > > > > > >
