Hey Tao, other than high latency between the brokers and the consumer, I'm not sure what would cause this. Can you turn on debug logging and run again? I'm looking for any connection problems or metadata/fetch request errors. And I have to ask a dumb question, how do you know that more messages are available? Are you monitoring the consumer's lag?
-Jason On Tue, Dec 1, 2015 at 10:07 AM, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > Thanks Tao, it worked. > I also played around with my test setting trying to replicate your results, > using default settings. But als long as the poll timeout is set to 100ms or > larger the only time-out I get are near the start and near the end (when > indeed there is nothing to consume). This is with a producer putting out > 1000 messages a second. Maybe the load of the producer your using is not > constant? Maybe you could run a test with the > org.apache.kafka.tools.ProducerPerformance class to see if it makes a > difference? > > On Tue, Dec 1, 2015 at 11:35 AM tao xiao <xiaotao...@gmail.com> wrote: > > > Gerard, > > > > In your case I think you can set fetch.min.bytes=1 so that the server > will > > answer the fetch request as soon as a single byte of data is available > > instead of accumulating enough messages. > > > > But in my case is I have plenty of messages in broker and I am sure the > > size of total message are much larger than the default setting which is > > 1024 bytes but still the consumer doesn't return messages for every poll. > > > > > > On Tue, 1 Dec 2015 at 18:29 Gerard Klijs <gerard.kl...@dizzit.com> > wrote: > > > > > I was experimenting with the timeout setting, but as long as messages > are > > > produced and the consumer(s) keep polling I saw little difference. I > did > > > see for example that when producing only 1 message a second, still it > > > sometimes wait to get three messages. So I also would like to know if > > there > > > is a faster way. > > > > > > On Tue, Dec 1, 2015 at 10:35 AM tao xiao <xiaotao...@gmail.com> wrote: > > > > > > > Hi team, > > > > > > > > I am using the new consumer with broker version 0.9.0. I notice that > > > > poll(time) occasionally returns 0 message even though I have enough > > > > messages in broker. The rate of returning 0 message is quite high > like > > 4 > > > > out of 5 polls return 0 message. It doesn't help by increasing the > poll > > > > timeout from 300ms to 1 second. are there any configurations that I > can > > > > tune to fetch data as quickly as possible? > > > > > > > > Both consumer and broker configs are default > > > > > > > > > >
