Franco, I think this is a mis-documentation of "poll(0)": https://issues.apache.org/jira/browse/KAFKA-3044
As for your case, I would suggest trying with poll(x) where x is reasonably large, it does not mean "always wait for that among of time", but "wait for at most approximately that among of time, and return as long as there are some data fetched". Guozhang On Wed, Dec 30, 2015 at 11:16 AM, Dana Powers <dana.pow...@gmail.com> wrote: > A few thoughts from a non-expert: > > connections are also processed asynchronously in the poll loop. If you are > not enabling any timeout, you may be seeing a few initial iterations spent > on setting up the channel connections. Also you probably need a few loop > iterations to get through an initial metadata request / response. > > also, if I recall, records should be returned in batches per > topic-partition; not one-by-one. So if/when records are ready, you would > get as many as were received via completed FetchRequests -- depends on > message size and fetch configs max.partition.fetch.bytes, fetch.min.bytes, > and fetch.max.wait.ms. So you shouldn't expect to poll 500x. > > I'd suggest using a small, but non-zero timeout when polling. 100ms is used > in the docs quite a bit. > > -Dana > > On Wed, Dec 30, 2015 at 10:03 AM, Franco Giacosa <fgiac...@gmail.com> > wrote: > > > Hi, > > > > I am running kafka 0.9.0 locally. > > > > I am having a particular situation in the following scenario. > > > > (1) 1 Producer inserts 500 records (300bytes each aprox) to 1 topic 0 > > partition (or 1 as you prefer) > > (2) After the producer finished inserting the 500 records, 1 Consumer > reads > > in a loop from this topic with consumer.poll(0) > > and max.partition.fetch.bytes=500, sometimes that call brings records and > > something the loop has to go over a few times until it brings something. > > Can someone explain me why it doesn't fetch a record each time that it > > polls? can a poll operation affect another poll operation? > > why if I've inserted 500 records I have to poll more than 500 times? > > > > I have tried using poll(0), because in the documentation it says, "if 0, > > returns with any records that are available now". > > > > Thanks > > > -- -- Guozhang
