Ben,
It could also be related to how you initialize auto.offset.reset. In unit
tests, you generally want to set it to 'smallest' to avoid race conditions
between producing and consuming.....
Jason
On Wed, May 20, 2015 at 2:32 PM, Padgett, Ben <bpadg...@illumina.com> wrote:
> Thanks for the detailed explanation. I was simply testing Kafka for the
> first time with a few "throw away" unit tests to learn it works and was
> curious why I was receiving that behavior.
> ________________________________________
> From: Jay Kreps [jay.kr...@gmail.com]
> Sent: Wednesday, May 20, 2015 10:29 AM
> To: users@kafka.apache.org
> Subject: Re: consumer poll returns no records unless called more than
> once, why?
>
> Hey Ben,
>
> The consumer actually doesn't promise to return records on any given poll()
> call and even in trunk it won't return records on the first call likely.
>
> Internally the reason is that it basically does one or two rounds of
> non-blocking actions and then returns. This could include things like
> communicating with the co-ordinator, establishing connections, sending
> fetch requests, etc.
>
> I guess the question is whether this behavior is confusing or not. In
> general there is no guarantee that you will have data ready, or that if you
> do you will be assigned a partition to consume from within your timeout. So
> assuming that poll will always return data is wrong.
>
> However with a little effort we could potentially wrap the poll call so
> that rather than return it would always attempt to wait the full timeout
> potentially doing multiple internal polls. This doesn't guarantee it would
> return data but would reduce the likelihood when data was ready.
>
> I'm not sure if that is actually a good idea vs just documenting this a
> little better in the javadoc.
>
> -Jay
>
> On Wed, May 20, 2015 at 10:12 AM, Guozhang Wang <wangg...@gmail.com>
> wrote:
>
> > Hello Ben,
> >
> > This Java consumer client was still not mature in 0.8.2.0 and lots of bug
> > fixes have been checked in since then.
> >
> > I just test your code with trunk's consumer and it does not illustrate
> this
> > problem. Could you try the same on your side and see if this issue goes
> > away?
> >
> > Guozhang
> >
> > On Wed, May 20, 2015 at 9:49 AM, Padgett, Ben <bpadg...@illumina.com>
> > wrote:
> >
> > > I am using Kafka v0.8.2.0
> > > ________________________________________
> > > From: Guozhang Wang [wangg...@gmail.com]
> > > Sent: Wednesday, May 20, 2015 9:41 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: consumer poll returns no records unless called more than
> > > once, why?
> > >
> > > Hello Ben,
> > >
> > > Which version of Kafka are you using with this consumer client?
> > >
> > > Guozhang
> > >
> > > On Wed, May 20, 2015 at 9:03 AM, Padgett, Ben <bpadg...@illumina.com>
> > > wrote:
> > >
> > > > //this code
> > > >
> > > > Properties consumerProps = new Properties();
> > > > consumerProps.put("bootstrap.servers", "localhost:9092");
> > > >
> > > >
> > > > //without deserializer it fails, which makes sense. the
> > > > documentation however doesn't show this
> > > > consumerProps.put("key.deserializer",
> > > > "org.apache.kafka.common.serialization.StringDeserializer");
> > > > consumerProps.put("value.deserializer",
> > > > "org.apache.kafka.common.serialization.StringDeserializer");
> > > >
> > > >
> > > > //why is serializer required? without this it fails to return
> > > > results when calling poll
> > > > consumerProps.put("key.serializer",
> > > > "org.apache.kafka.common.serializers.StringSerializer");
> > > > consumerProps.put("value.serializer",
> > > > "org.apache.kafka.common.serializers.StringSerializer");
> > > >
> > > >
> > > > consumerProps.put("group.id", "test");
> > > > consumerProps.put("enable.auto.commit", "true");
> > > > consumerProps.put("auto.commit.interval.ms", "1000");
> > > > consumerProps.put("session.timeout.ms", "30000");
> > > >
> > > > org.apache.kafka.clients.consumer.KafkaConsumer<String,
> String>
> > > > consumer = new
> org.apache.kafka.clients.consumer.KafkaConsumer<String,
> > > > String>(consumerProps);
> > > >
> > > > TopicPartition topicPartition = new
> > > > TopicPartition("project-created", 0);
> > > > consumer.subscribe(topicPartition);
> > > >
> > > > consumer.seeekToBeginning(topicPartition);
> > > >
> > > > //each scenerio code goes here
> > > >
> > > >
> > > >
> > > > I have a scenerio where it returns records and a scenerio
> where
> > > no
> > > > records are returned. Could anyone provide insight on why this
> happens?
> > > >
> > > >
> > > >
> > > > //without a loop consumer.poll(100) returns no records
> > > > //after poll is called a second time it returns records
> > > > boolean run = true;
> > > > while (run) {
> > > > ConsumerRecords<String, String> records =
> > consumer.poll(100);
> > > > }
> > > >
> > > >
> > > >
> > > > //why would this return zero records?
> > > > ConsumerRecords<String, String> records = consumer.poll(100);
> > > >
> > > >
> > > > //This is to show that there are records for topic "project-created"
> > > >
> > > > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
> > > > project-created --from-beginning
> > > > 3a543cf4-13e4-42b6-8c72-54b733228c75
> > > > c298d41e-4ae5-4f4f-933e-f745969aaf98
> > > > 2036ef69-9694-4ee8-a40c-50cddf982edb
> > > > 08d90698-8c75-4c4b-8ce6-b0b12d3de0d6
> > > > 249d0e8c-cd38-41b0-ae7f-bdfa7fa8f76e
> > > > 84d87e45-cd10-4797-bfe3-d540e26e56cc
> > > > a8c74f78-cb7a-4c74-873b-a0e587af646c
> > > > 42eeff6e-22fd-40c4-96b4-91a3df356ec7
> > > > bef622b6-3ac1-489b-9837-ab9c2b0399d1
> > > > 03e9d567-fadf-46dc-8097-4bc327a0942e
> > > > df5d93d6-30de-494a-8d82-45cd8b4fc785
> > > > eb5a194f-083d-4a0f-bbb7-155056acd929
> > > > 192dfbb7-4f4f-4550-a1b4-ab9bc83be825
> > > > b7845e7e-b477-476d-b115-b0a9d076c52e
> > > > f8ea2d0f-bc76-44a8-86ef-2b3e0adf7755
> > > > 8ee28e0c-5a8e-47c8-a939-f5dc6d7be3f9
> > > > fbe30ae3-c383-4e27-8f70-f3c63ed82fed
> > > > 50fa9166-cc0d-4d12-8d86-c922062519cd
> > > > 50ceb437-2556-4a9a-8a13-93bfc130e914
> > > > f1f5ad0b-7739-47fd-be57-ac093f728004
> > > > 0df1a923-92d9-416a-861a-d65590f7e4c2
> > > > 849e5e2b-3803-4a6e-86fc-64fe80d81268
> > > > 163d9c1c-6224-4dad-a23a-1c90c9791e1f
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
