Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. Here's a runnable gist illustrating the problem: https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on localhost:9092)
Scenario description: First, a producer writes 500000 elements into a topic Then, a consumer starts to read, polling in a loop. When "max.partition.fetch.bytes" is set to a relatively small value, each "consumer.poll()" returns a batch of messages. If this value is left as default, the output tends to look like this: Poll returned 13793 elements Poll returned 13793 elements Poll returned 13793 elements Poll returned 13793 elements Poll returned 0 elements Poll returned 0 elements Poll returned 0 elements Poll returned 0 elements Poll returned 13793 elements Poll returned 13793 elements As we can see, there are weird "gaps" when poll returns 0 elements for some time. What is the reason for that? Maybe there are some good practices about setting "max.partition.fetch.bytes" which I don't follow? Bests, Chris
