Hi Fares,

What is the rate of offset commits for the group?
How often do you need to commit offsets for consumers in this group?

Thanks,
Alexandre

Le mar. 9 mai 2023 à 18:34, Fares Oueslati <oueslati.fa...@gmail.com> a écrit :
>
> Hello Richard,
>
> Thank you for your answer.
>
> Upon examining the `__consumer_offsets` topic, it seems that all commit
> messages for a given consumer `group.id` go to the same partition.
> So, there's nothing much to do if we have a dominant consumer group reading
> from all topics.
>
> The only solution would be to split it in multiple consumer groups reading
> from different subsets of topics.
>
> Best Regards,
> Fares
>
> On Mon, May 1, 2023 at 11:07 AM Richard Bosch <richard.bo...@axual.com>
> wrote:
>
> > Hi Fares,
> >
> > You're right in your description of the contents of the __consumer_offsets
> > topic, and how they are stored.
> > The most common reason for an uneven load on the consumer offsets are.
> >
> > 1. Configuration of offset commits in the client
> > 2. Load on topic being consumed
> >
> > If a topic has 10 partitions, and the producer produces records with a key
> > and a partitioner based on the key hash, then it can happen that one or
> > more partitions get more records than the others. Just because several keys
> > are more often used than others.
> > Now the consumer needs to read more records from those partitions.
> > If the consumer commits offsets at a time interval or after an N amount of
> > records consumed, it follows that this results in more offset commits for
> > the topic partitions containing more records.
> >
> > You might want to check the load on the topic partitions being consumed to
> > confirm this is the case.
> > Unfortunately, I do not have an easy answer on how to remedy that problem.
> >
> > You can check what the offset commit settings are for your application, and
> > if you can update the logic to reflect the higher load.
> > If the trigger is based on time or nrOfRecords consumed you can alter these
> > values to make sure they aren't triggered that often.
> >
> > You can also start monitoring and updating the cluster with a preferred
> > leader election setting for your partitions using CruiseControl to minimize
> > the load on your brokers.
> >
> > I do recommend to use keys with the hash based partitioner only if the
> > order of messages for those keys MUST be guaranteed, else you can use a
> > different Partitioner.
> > Then there will be a more uniform distribution of records on the topic, and
> > a better distribution of offset commits.
> >
> > Do note that there is an issue with the UniformStickyPartitioner that is
> > being worked on right now, see
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner
> >
> > I hope this helps
> >
> > Kind regards,
> >
> >
> > Richard Bosch
> >
> > Developer Advocate
> >
> > Axual BV
> >
> > E : richard.bo...@axual.com
> > M : +31 6 11 850 846
> > W : www.axual.com
> >
> >
> > On Thu, Apr 27, 2023 at 4:19 PM Fares Oueslati <oueslati.fa...@gmail.com>
> > wrote:
> >
> > > Hello Kafka users,
> > >
> > > I’m facing an issue with a Kafka cluster, specifically with the
> > > __consumer_offsets topic.
> > > There seems to be an imbalance in the number of commit messages across
> > its
> > > partitions. Most of the commit messages are concentrated in a single
> > > partition, which is causing high CPU usage on the broker handling that
> > > partition.
> > > I have already verified that the topic partitions’ leaders are
> > > well-balanced across the six brokers.
> > > However, a specific consumer group (The largest one, with many members
> > > consuming from multiple topics, based on Spring Kafka) generates a large
> > > number of commit messages, and they all end up in the same partition #37.
> > > My understanding is that, by default, all commit messages sent by a
> > > particular consumer group for a specific topic partition are directed to
> > a
> > > single partition of the __consumer_offsets topic, determined by hashing
> > the
> > > consumer group id and the topic partition. In our case, this default
> > > partitioning strategy seems to be causing the imbalance, even though I
> > > don’t understand why exactly.
> > > Could you please help me understand why there’s such an imbalance in the
> > > number of messages across the __consumer_offsets partitions and why the
> > > large number of commit messages from the large consumer group are not
> > > spread well across the partitions of the __consumer_offsets topic? Are
> > > there any recommendations or best practices to address this issue?
> > >
> > > Any guidance would be greatly appreciated.
> > >
> > > Best Regards,
> > > Fares
> > >
> >

Reply via email to