[
https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622015#comment-15622015
]
Yiquan Zhou commented on KAFKA-4348:
------------------------------------
Did you get any "polled 0 records" before polling all the records? Because
sometimes the consumer did poll all the records without interruption like what
you got, but only after several seconds polling 0 record. This happens often
right after the start of the server. But if you run the test for 4-5 more times
you might get the same result as I did.
If I set "enable.auto.commit" to true, here is an output that I got:
Mon Oct 31 11:25:39 CET 2016 polled 0 records // first poll
Mon Oct 31 11:25:40 CET 2016 polled 0 records
Mon Oct 31 11:25:41 CET 2016 polled 0 records
Mon Oct 31 11:25:42 CET 2016 polled 0 records
Mon Oct 31 11:25:43 CET 2016 polled 0 records
Mon Oct 31 11:25:43 CET 2016 polled 27171 records
Mon Oct 31 11:25:43 CET 2016 polled 26886 records
Mon Oct 31 11:25:43 CET 2016 polled 26886 records
Mon Oct 31 11:25:44 CET 2016 polled 0 records
Mon Oct 31 11:25:45 CET 2016 polled 0 records
Mon Oct 31 11:25:46 CET 2016 polled 0 records
Mon Oct 31 11:25:47 CET 2016 polled 0 records
Mon Oct 31 11:25:48 CET 2016 polled 0 records
Mon Oct 31 11:25:48 CET 2016 polled 26690 records
Mon Oct 31 11:25:49 CET 2016 polled 26214 records
Mon Oct 31 11:25:49 CET 2016 polled 26214 records
Mon Oct 31 11:25:49 CET 2016 polled 26214 records
Mon Oct 31 11:25:49 CET 2016 polled 13725 records
Mon Oct 31 11:25:50 CET 2016 polled 0 records
Mon Oct 31 11:25:51 CET 2016 polled 0 records
Mon Oct 31 11:25:52 CET 2016 polled 0 records
> On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on
> Kafka server
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-4348
> URL: https://issues.apache.org/jira/browse/KAFKA-4348
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.1
> Environment: Mac OS X EI Capitan, Java 1.8.0_111
> Reporter: Yiquan Zhou
> Labels: consumer, mac, polling
>
> Steps to reproduce:
> 1. start the zookeeper and kafka server using the default properties from the
> distribution:
> $ bin/zookeeper-server-start.sh config/zookeeper.properties
> $ bin/kafka-server-start.sh config/server.properties
> 2. create a Kafka consumer using the Java API KafkaConsumer.poll(long
> timeout). It polls the records from the server every second (timeout set to
> 1000) and prints the number of records polled. The code can be found here:
> https://gist.github.com/yiquanzhou/a94569a2c4ec8992444c83f3c393f596
> 3. use bin/kafka-verifiable-producer.sh to generate some messages:
> $ bin/kafka-verifiable-producer.sh --topic connect-test --max-messages 200000
> --broker-list localhost:9092
> wait until all 200k messages are generated and sent to the server.
> 4. Run the consumer Java code. In the output console of the consumer, we can
> see that the consumer starts to poll some records, then it polls 0 records
> for several seconds before polling some more. like this:
> polled 27160 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26886 records
> polled 26886 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26701 records
> polled 26214 records
> The bug slows down the consumption of messages a lot. And in our use case,
> the consumer wrongly assumes that all messages are read from the topic.
> It is only reproducible on Mac OS X but neither on Linux nor Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
