Yes, there has been a few issues reported related to message latency. We are fixing that in https://issues.apache.org/jira/browse/KAFKA-1430
Thanks, Jun On Wed, May 7, 2014 at 7:30 AM, Simon Cooper < [email protected]> wrote: > I've got a very strange bug doing a long poll using SimpleConsumer on > 0.8.1.1. The test app I've got (attached) uses a thread to do a long poll > of a topic, and then sends single messages to that queue, one at a time, > waiting for the poll to return the message before sending the next one. > > The problem is that when a message is sent to the queue, the long poll > returns immediately, but that does not contain the just-sent message. So > the long poll restarts. Sometime this second poll returns immediately with > the sent message, but sometimes it waits for the poll to timeout before > returning the message. This second behaviour happens much more on a > replicated topic, but is also reproducible with the test VMs on > https://github.com/stealthly/scala-kafka. > > A sample log file is below. The long poll timeout is 5000ms. Messages 1 > and 2 are delayed, messages 0 and 2 aren't. Every message requires two > polls to actually be received: > > 410 [pool-1-thread-1] INFO Making request from offset 110 > 1457 [main] INFO Sending 0 > 1553 [pool-1-thread-1] WARN Received no messages > 1553 [pool-1-thread-1] INFO Making request from offset 110 > 1560 [pool-1-thread-1] INFO Received 0 > 1561 [pool-1-thread-1] INFO Making request from offset 111 > 1561 [main] INFO Got response for 0 in 20ms > 1561 [main] INFO Sending 1 > 1565 [pool-1-thread-1] WARN Received no messages > 1565 [pool-1-thread-1] INFO Making request from offset 111 > 6567 [pool-1-thread-1] INFO Received 1 > 6567 [pool-1-thread-1] INFO Making request from offset 112 > 6568 [main] INFO Got response for 1 in 5004ms > 6568 [main] INFO Sending 2 > 6574 [pool-1-thread-1] WARN Received no messages > 6574 [pool-1-thread-1] INFO Making request from offset 112 > 11577 [pool-1-thread-1] INFO Received 2 > 11577 [pool-1-thread-1] INFO Making request from offset 113 > 11578 [main] INFO Got response for 2 in 5006ms > 11578 [main] INFO Sending 3 > 11584 [pool-1-thread-1] WARN Received no messages > 11584 [pool-1-thread-1] INFO Making request from offset 113 > 11585 [pool-1-thread-1] INFO Received 3 > 11586 [pool-1-thread-1] INFO Making request from offset 114 > 11586 [main] INFO Got response for 3 in 5ms > > Ideally, the message sent would be returned immediately by the first long > poll. But the obvious bug is the artificial delay introduced by the message > not being returned until the long poll times out, when it should return > immediately (as the minimum message size is 1 byte, and there's a 4 byte > message available). > > This looks like a bug in kafka, but I may be using the consumer API > incorrectly... > > Thanks, > SimonC >
