[
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439670#comment-15439670
]
Dustin Cote commented on KAFKA-3129:
------------------------------------
What I'm seeing is that we are faking the callback
{code}org.apache.kafka.clients.producer.internals.Sender#handleProduceResponse{code}
for the case where acks=0. This is a problem because the callback gets
generated when we do
{code}org.apache.kafka.clients.producer.internals.Sender#createProduceRequests{code}
in the run loop but the actual send happens a bit later. When close() comes
in that window between createProduceRequests and the send, you get messages
that are lost. Funny thing is that if you slow down call stack a bit by
turning on something like strace, the issue goes away so it's hard to tell
which layer exactly is buffering the requests.
So my question is, do we want to risk a small performance hit for all producers
to be able to guarantee all messages with acks=0 actually make it out of the
producer knowing full well that they aren't going to be verified to have made
it to the broker? I personally don't feel it's worth the extra locking
complexity and could be documented known durability issue when you aren't using
durability settings. If we go that route, I feel like the console producer
should have acks=1 by default. That way, users who are getting started with
the built-in tools have a cursory durability guarantee and can tune for
performance instead. What do you think [~ijuma] and [~vahid]?
> Console producer issue when request-required-acks=0
> ---------------------------------------------------
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.9.0.0, 0.10.0.0
> Reporter: Vahid Hashemian
> Assignee: Neha Narkhede
> Attachments: kafka-3129.mov, server.log.abnormal.txt,
> server.log.normal.txt
>
>
> I have been running a simple test case in which I have a text file
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test <
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it
> goes all the way to 1,000,000. But, in other cases, it stops short, usually
> at 999,735.
> I tried running another console consumer on another machine and both
> consumers behave the same way. I can't see anything related to this in the
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am
> getting a similar behavior. When the consumer does not receive all the 10,000
> messages it usually stops at 9,864.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
