Our consumer group isn't doing anything stateful and we've seen this behavior for existing groups as well. It seems like timing could be an issue, thanks for the information.
On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wushuja...@gmail.com> wrote: > Bryan, > > This sounds related to > https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance > and https://issues.apache.org/jira/browse/KAFKA-4925. > > -James > > > On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bjb...@gmail.com> wrote: > > > > The topics already exist prior to starting any of the consumers > > > > On Mon, Jun 12, 2017 at 9:35 PM J Pai <jai.forums2...@gmail.com> wrote: > > > >> When are the topics on which these consumer groups consume, created? > >> > >> -Jaikiran > >> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bjb...@gmail.com> wrote: > >> > >> Hi everyone, > >> > >> We are currently experiencing slow startup times for our consumer groups > >> (16-32 processes for a hundred or more partitions) in the range of > minutes > >> (3-15 minutes), where little to no messages are consumed before suddenly > >> everything just starts working at full speed. > >> > >> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading > to > >> Kafka 0.10.2.1. We also using the newer kafka consumer API and group > >> management on a simple Apache Storm topology. We don't make use of > Storm's > >> kafka spout but instead wrote a simple one ourselves. > >> > >> Using the kafka AdminClient I was able to poll for consumer group > summary > >> information. What I've found is that the group seems to sit > >> in PreparingRebalance state for minutes before finally becoming Stable > >> which then everything starts processing quickly. I've also enabled debug > >> logging around the consumer's coordinator classes but didn't see > anything > >> to indicate the issue. > >> > >> I'm hoping that just upgrading to 0.10 or tweaking how we use our > consumer > >> in Apache Storm is the problem but are there any pointers on things I > >> should look at or try? > >> > >> Bryan > >> > >> > >
