1. We use cruise-control to actively balance the partitions across all brokers. So point 1 could be ruled out. 2. I am not sure how much this would impact the broker, as we do have some exceptionally large partitions around. I have to check to know if they live on the aforementioned broker. So far I don't see there is strong correlation between total producer / consumer byte rates with CPU spikes on this broker.
Tiansu Yu Engineer Data Ingestion & Streaming Klarna Bank AB German Branch Chausseestraße 117 10115 Berlin Tel: +49 221 669 501 00 klarna.de Klarna Bank AB, German Branch Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B USt-Nr.: DE 815 867 324 Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit Hauptsitz in Stockholm, Schw. Gesellschaftsregister 556737-0431 Verwaltungsratsvorsitzender: Michael Moritz Geschäftsführender Direktor: Sebastian Siemiatkowski Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen > On 9. Aug 2023, at 12:05, sunil chaudhari <sunilmchaudhar...@gmail.com> wrote: > > Hi I can guess two problems here. > 1. Either too many partition’s concentrated on this broker compared to > other broker > 2. The partitions on this broker might have larger size as compared to the > parition on other brokers > > please chech if all brokers are evenly balanced in terms of number of > partitions and the total topic size on each broker. > > On Wed, 9 Aug 2023 at 1:29 PM, Tiansu Yu <tiansu...@klarna.com.invalid > <mailto:tiansu...@klarna.com.invalid>> > wrote: > >> Hi Kafka community, >> >> We have an issue with our Kafka cluster from time to time, that a single >> (one and only one) broker (leader) in the cluster reaches 100% CPU >> utilisation. We could not see any apparent issue from the metrics. There is >> no heap memory usage increase, no excessive connections made on the broker, >> no misbehaving producers and consumers trying to dump or load excessively >> during these periods. The only difference we could see is that thread usage >> decreases during these period. Despite the problem, the service is still >> available (understandable from Kafka's perspective.) >> >> We are trying to understand what else might be the cause of the issue and >> how we can mitigate them. >> >> Tiansu Yu >> Engineer >> Data Ingestion & Streaming >> >> Klarna Bank AB German Branch >> Chausseestraße 117 >> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g> >> 10115 Berlin >> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g> >> Tel: +49 221 669 501 00 >> klarna.de >> >> Klarna Bank AB, German Branch >> Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B >> USt-Nr.: DE 815 867 324 >> Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit >> Hauptsitz in Stockholm, >> Schw. Gesellschaftsregister 556737-0431 >> Verwaltungsratsvorsitzender: Michael Moritz >> Geschäftsführender Direktor: Sebastian Siemiatkowski >> Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen
smime.p7s
Description: S/MIME cryptographic signature
