Hi Fares, What is the rate of offset commits for the group? How often do you need to commit offsets for consumers in this group?
Thanks, Alexandre Le mar. 9 mai 2023 à 18:34, Fares Oueslati <oueslati.fa...@gmail.com> a écrit : > > Hello Richard, > > Thank you for your answer. > > Upon examining the `__consumer_offsets` topic, it seems that all commit > messages for a given consumer `group.id` go to the same partition. > So, there's nothing much to do if we have a dominant consumer group reading > from all topics. > > The only solution would be to split it in multiple consumer groups reading > from different subsets of topics. > > Best Regards, > Fares > > On Mon, May 1, 2023 at 11:07 AM Richard Bosch <richard.bo...@axual.com> > wrote: > > > Hi Fares, > > > > You're right in your description of the contents of the __consumer_offsets > > topic, and how they are stored. > > The most common reason for an uneven load on the consumer offsets are. > > > > 1. Configuration of offset commits in the client > > 2. Load on topic being consumed > > > > If a topic has 10 partitions, and the producer produces records with a key > > and a partitioner based on the key hash, then it can happen that one or > > more partitions get more records than the others. Just because several keys > > are more often used than others. > > Now the consumer needs to read more records from those partitions. > > If the consumer commits offsets at a time interval or after an N amount of > > records consumed, it follows that this results in more offset commits for > > the topic partitions containing more records. > > > > You might want to check the load on the topic partitions being consumed to > > confirm this is the case. > > Unfortunately, I do not have an easy answer on how to remedy that problem. > > > > You can check what the offset commit settings are for your application, and > > if you can update the logic to reflect the higher load. > > If the trigger is based on time or nrOfRecords consumed you can alter these > > values to make sure they aren't triggered that often. > > > > You can also start monitoring and updating the cluster with a preferred > > leader election setting for your partitions using CruiseControl to minimize > > the load on your brokers. > > > > I do recommend to use keys with the hash based partitioner only if the > > order of messages for those keys MUST be guaranteed, else you can use a > > different Partitioner. > > Then there will be a more uniform distribution of records on the topic, and > > a better distribution of offset commits. > > > > Do note that there is an issue with the UniformStickyPartitioner that is > > being worked on right now, see > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > > > I hope this helps > > > > Kind regards, > > > > > > Richard Bosch > > > > Developer Advocate > > > > Axual BV > > > > E : richard.bo...@axual.com > > M : +31 6 11 850 846 > > W : www.axual.com > > > > > > On Thu, Apr 27, 2023 at 4:19 PM Fares Oueslati <oueslati.fa...@gmail.com> > > wrote: > > > > > Hello Kafka users, > > > > > > I’m facing an issue with a Kafka cluster, specifically with the > > > __consumer_offsets topic. > > > There seems to be an imbalance in the number of commit messages across > > its > > > partitions. Most of the commit messages are concentrated in a single > > > partition, which is causing high CPU usage on the broker handling that > > > partition. > > > I have already verified that the topic partitions’ leaders are > > > well-balanced across the six brokers. > > > However, a specific consumer group (The largest one, with many members > > > consuming from multiple topics, based on Spring Kafka) generates a large > > > number of commit messages, and they all end up in the same partition #37. > > > My understanding is that, by default, all commit messages sent by a > > > particular consumer group for a specific topic partition are directed to > > a > > > single partition of the __consumer_offsets topic, determined by hashing > > the > > > consumer group id and the topic partition. In our case, this default > > > partitioning strategy seems to be causing the imbalance, even though I > > > don’t understand why exactly. > > > Could you please help me understand why there’s such an imbalance in the > > > number of messages across the __consumer_offsets partitions and why the > > > large number of commit messages from the large consumer group are not > > > spread well across the partitions of the __consumer_offsets topic? Are > > > there any recommendations or best practices to address this issue? > > > > > > Any guidance would be greatly appreciated. > > > > > > Best Regards, > > > Fares > > > > >
