As Chandrajeet said, the default behavior is to hash on the key to assign the message to a partition. You may be able to distribute the data more evenly across the brokers by changing the partition strategy in the producer. It *is* possible to change the default behavior to, say, round-robin. That might be an option if per-key ordering isn't a requirement.
On Mon, Feb 10, 2020 at 12:48 PM Dylan Martin <dmar...@istreamplanet.com> wrote: > > Oooh, that sounds like our situation. > > Is there a way to avoid this with kafka configuration? My access to the > messages and consumers is limited. > ________________________________ > From: Chandrajeet Padhy <chandraj...@truefort.com> > Sent: Monday, February 10, 2020 9:52 AM > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: RE: Why would all consumers pull from the same partition? > > [EXTERNAL E-MAIL] > > Partition is decided based on message record key. If it's same, it will > hit the same partition. > > -----Original Message----- > From: Dylan Martin <dmar...@istreamplanet.com> > Sent: Monday, February 10, 2020 12:35 PM > To: users@kafka.apache.org > Subject: Why would all consumers pull from the same partition? > > I have a cluster of 20'ish brokers. One topic has 60'ish consumers and > 100 partitions, but the consumers all seem to be hitting the same broker, > which makes me think they're all hitting the same partition. > > What would cause that? I assume I've configured something wrong. > > Thanks! > > -Dylan > > The information contained in this email message, and any attachment > thereto, is confidential and may not be disclosed without the sender's > express permission. If you are not the intended recipient or an employee or > agent responsible for delivering this message to the intended recipient, > you are hereby notified that you have received this message in error and > that any review, dissemination, distribution or copying of this message, or > any attachment thereto, in whole or in part, is strictly prohibited. If you > have received this message in error, please immediately notify the sender > by telephone, fax or email and delete the message and all of its > attachments. Thank you. > > The information contained in this email message, and any attachment > thereto, is confidential and may not be disclosed without the sender's > express permission. If you are not the intended recipient or an employee or > agent responsible for delivering this message to the intended recipient, > you are hereby notified that you have received this message in error and > that any review, dissemination, distribution or copying of this message, or > any attachment thereto, in whole or in part, is strictly prohibited. If you > have received this message in error, please immediately notify the sender > by telephone, fax or email and delete the message and all of its > attachments. Thank you. >
