You can also set fetch.min.bytes, It will be a trade-off between getting the messages as fast as possible, with fetch.min.bytes=0 and the poll at 0. This will result in a lot of empty returns, and a lot of io overhead as compared to getting a lot of massages in one go. If you for example set the fetch.min.bytes to max.int, and the poll timeout to 60000 you might only get one big bunch of records every minute.
On Tue, Dec 29, 2015 at 8:34 AM Praveen Devarao (JIRA) <j...@apache.org> wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073597#comment-15073597 > ] > > Praveen Devarao commented on KAFKA-3044: > ---------------------------------------- > > Hi [~guozhang] and [~jkreps] > > OK. > > Also, could we recommend a value for timeout [based on some assumed > factors]? > > Thanks > > Praveen > > > Consumer.poll doesnot return messages when poll interval is less > > ---------------------------------------------------------------- > > > > Key: KAFKA-3044 > > URL: https://issues.apache.org/jira/browse/KAFKA-3044 > > Project: Kafka > > Issue Type: Bug > > Components: clients > > Affects Versions: 0.9.0.0 > > Reporter: Praveen Devarao > > Assignee: Jason Gustafson > > Fix For: 0.9.0.1 > > > > > > When seeking to particular position in consumer and starting poll with > timeout param 0 the consumer does not come back with data though there is > data published via a producer already. If the timeout is increased slowly > in chunks of 100ms then at 700ms value the consumer returns back the record > on first call to poll. > > Docs [ > http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long)] > for poll reads if timeout is 0 then data will be returned immediately but > the behaviour seen is that data is not returned. > > The test code I am using can be found here > https://gist.github.com/praveend/013dcab01ebb8c7e2f2d > > I have created a topic with data published as below and then running the > test program [ConsumerPollTest.java] > > $ bin/kafka-topics.sh --create --zookeeper localhost:2181 > --replication-factor 1 --partitions 1 --topic mytopic > > $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic > mytopic > > Hello > > Hai > > bye > > $ java ConsumerPollTest > > I have published this 3 lines of data to kafka only once....later on I > just use the above program with different poll interval > > Let me know if I am missing anything and interpreting it wrongly. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
