Hi Gerard, I am not sure why min.fetch.bytes setting will cause a pause. Also, in 0.9 config, there is no min.fetch.bytes. The only bytes related setting is MAX_PARTITION_FETCH_BYTES_CONFIG. See: https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/ConsumerConfig.html
It definitely looks like a timeout is triggered somewhere. Anyone else run into this issue with 0.9 console consumer? On Tue, Jan 12, 2016 at 8:01 AM, Brice Dutheil <brice.duth...@gmail.com> wrote: > Hi Gerard, > > Why the fetch size should be correlated to the consumer stalling after x > messages. > > One can st the fetch size on a cassandra query, and yet there's no > "stalling", it's more or less just another "page". > > > > Cheers, > -- Brice > > On Tue, Jan 12, 2016 at 12:10 PM, Gerard Klijs <gerard.kl...@dizzit.com> > wrote: > > > Hi Suyog, > > It working as intended. You could set the property min.fetch.bytes to a > > small value to get less messages in each batch. Setting it to zero will > > probably mean you get one object with each batch, at least was the case > > when I tried, but I was producing and consuming at the same time. > > > > On Tue, Jan 12, 2016 at 3:47 AM Suyog Rao <suyog....@gmail.com> wrote: > > > > > Hi, I started with a clean install of 0.9 Kafka broker and populated a > > test > > > topic with 1 million messages. I then used the console consumer to read > > > from beginning offset. Using --new-consumer reads the messages, but it > > > stalls after every x number of messages or so, and then continues > again. > > It > > > is very batchy in its behaviour. If I go back to the old consumer, I am > > > able to stream the messages continuously. Am I missing a timeout > setting > > or > > > something? > > > > > > I created my own consumer in Java and call poll(0) in a loop, but I > still > > > get the same behaviour. This is on Mac OS X (yosemite) with java > version > > > "1.8.0_65". > > > > > > Any ideas? > > > > > > bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic > > > apache_logs --from-beginning --new-consumer > > > > > > bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic > > > apache_logs --from-beginning -zookeeper localhost:2181 > > > > > >
