The topics already exist prior to starting any of the consumers On Mon, Jun 12, 2017 at 9:35 PM J Pai <jai.forums2...@gmail.com> wrote:
> When are the topics on which these consumer groups consume, created? > > -Jaikiran > On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bjb...@gmail.com> wrote: > > Hi everyone, > > We are currently experiencing slow startup times for our consumer groups > (16-32 processes for a hundred or more partitions) in the range of minutes > (3-15 minutes), where little to no messages are consumed before suddenly > everything just starts working at full speed. > > I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to > Kafka 0.10.2.1. We also using the newer kafka consumer API and group > management on a simple Apache Storm topology. We don't make use of Storm's > kafka spout but instead wrote a simple one ourselves. > > Using the kafka AdminClient I was able to poll for consumer group summary > information. What I've found is that the group seems to sit > in PreparingRebalance state for minutes before finally becoming Stable > which then everything starts processing quickly. I've also enabled debug > logging around the consumer's coordinator classes but didn't see anything > to indicate the issue. > > I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer > in Apache Storm is the problem but are there any pointers on things I > should look at or try? > > Bryan > >
