Ok, I think I found the cause of the problem. The default value
of max_in_flight_requests_per_connection is 5 in Python producer. This
turns out too small for my environment and application. When this value is
reached and the producer tries several times and still fails, the message
is dropped. And
PS: The message dropping occurred intermittently, not all at the end. For
example, it is the 10th, 15th, 18th messages that are missing. It it were
all at the end, it would be understandable because I'm not using flush() to
force transmitting.
Bo
On Thu, Apr 28, 2016 at 10:15 AM, Bo Xu wrote:
I set up a simple Kafka configuration, with one topic and one partition. I
have a Python producer to continuously publish messages to the Kafka server
and a Python consumer to receive messages from the server. Each message is
about 10K bytes, far smaller than socket.request.max.bytes=104857600. Wha
