Absolutely; I created https://issues.apache.org/jira/browse/KAFKA-3627 and included fairly detailed steps to reproduce, including the consumer configuration. Thanks!
________________________________________ From: Liquan Pei <liquan...@gmail.com> Sent: Tuesday, April 26, 2016 3:51 AM To: users@kafka.apache.org Subject: Re: Consumer doesn't run delayed tasks while under load Hi Robert, I think the scenario you describe is possible. It would be nice if you can provide more information on how to reproduce the issue. Also, would you mind to file a JIRA to track the issue? In the JIRA, it would be nice if you can share the configurations of the consumer. Thanks, Liquan On Mon, Apr 25, 2016 at 1:55 PM, Underwood, Robert < robert.underw...@inin.com> wrote: > I've been investigating consumer group rebalances happening when I don't > think they should and have noticed an issue. In a nutshell, if a consumer > is receiving messages in response to every fetch request then it won't run > delayed tasks, most notably heartbeats and automatic commits, which in turn > will cause a rebalance. > > > Note that in the situation I'm describing the consumer is polling > regularly, well within the session timeout, so a rebalance is not expected. > > > In KafkaConsumer::pollOnce there is a check for fetched records, and if > there are records found then it skips running client.poll. Then up in > KafkaConsumer::poll if records are returned it initiates fetches and does a > quick poll, which won't run delayed tasks but will receive fetched > records. So if the fetch responses are coming in during every quick poll > the consumer gets in a state where it's never calling client.poll and > running delayed tasks (until it stops receiving records in response to its > fetches). > > > I can provide detailed reproduction steps if needs be. The key parameters > are that there must be at least 2 brokers involved, and the max fetch size > should be reduced, to limit the size of the fetch batches. > > > If anyone can verify what I'm seeing I'll create a bug, and if anyone has > any ideas on how to prevent this from happening I would appreciate them. > -- Liquan Pei Software Engineer, Confluent Inc
