1. We use cruise-control to actively balance the partitions across all brokers. 
So point 1 could be ruled out. 
2. I am not sure how much this would impact the broker, as we do have some 
exceptionally large partitions around. I have to check to know if they live on 
the aforementioned broker. So far I don't see there is strong correlation 
between total producer / consumer byte rates with CPU spikes on this broker. 

Tiansu Yu
Engineer
Data Ingestion & Streaming

Klarna Bank AB German Branch
Chausseestraße 117
10115 Berlin
Tel: +49 221 669 501 00
klarna.de

Klarna Bank AB, German Branch
Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B
USt-Nr.: DE 815 867 324  
Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit Hauptsitz in 
Stockholm,
Schw. Gesellschaftsregister 556737-0431
Verwaltungsratsvorsitzender: Michael Moritz
Geschäftsführender Direktor: Sebastian Siemiatkowski
Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen

> On 9. Aug 2023, at 12:05, sunil chaudhari <sunilmchaudhar...@gmail.com> wrote:
> 
> Hi I can guess two problems here.
> 1. Either too many partition’s concentrated on this broker compared to
> other broker
> 2. The partitions on this broker might have larger size as compared to the
> parition on other brokers
> 
> please chech if all brokers are evenly balanced in terms of number of
> partitions and the total topic size on each broker.
> 
> On Wed, 9 Aug 2023 at 1:29 PM, Tiansu Yu <tiansu...@klarna.com.invalid 
> <mailto:tiansu...@klarna.com.invalid>>
> wrote:
> 
>> Hi Kafka community,
>> 
>> We have an issue with our Kafka cluster from time to time, that a single
>> (one and only one) broker (leader) in the cluster reaches 100% CPU
>> utilisation. We could not see any apparent issue from the metrics. There is
>> no heap memory usage increase, no excessive connections made on the broker,
>> no misbehaving producers and consumers trying to dump or load excessively
>> during these periods. The only difference we could see is that thread usage
>> decreases during these period. Despite the problem, the service is still
>> available (understandable from Kafka's perspective.)
>> 
>> We are trying to understand what else might be the cause of the issue and
>> how we can mitigate them.
>> 
>> Tiansu Yu
>> Engineer
>> Data Ingestion & Streaming
>> 
>> Klarna Bank AB German Branch
>> Chausseestraße 117
>> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g>
>> 10115 Berlin
>> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g>
>> Tel: +49 221 669 501 00
>> klarna.de
>> 
>> Klarna Bank AB, German Branch
>> Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B
>> USt-Nr.: DE 815 867 324
>> Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit
>> Hauptsitz in Stockholm,
>> Schw. Gesellschaftsregister 556737-0431
>> Verwaltungsratsvorsitzender: Michael Moritz
>> Geschäftsführender Direktor: Sebastian Siemiatkowski
>> Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Reply via email to