Re: High CPU Usage on Brokers

2020-01-13 Thread Ismael Juma
You can take a profile with Java Flight Recorder if you use Java 11 or using async profiler otherwise. See below for the latter: https://issues.apache.org/jira/browse/KAFKA-9339?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17013400#comment-17013400 It's

Re: High CPU Usage on Brokers

2020-01-12 Thread Navneeth Krishnan
Hi Ismael, We were previously running on 0.10.2.1 with 8 brokers running around 80% CPU. But now we have upgraded to 2.3 with 16 brokers. It's the same message rate, topics, producers and consumers but the CPU is still >80%. How can we troubleshoot to find where exactly is the problem? Thanks On

Re: High CPU Usage on Brokers

2020-01-12 Thread Navneeth Krishnan
Hi Lisheng, Here are the answers to your questions. do you set sun.security.jgss.native = true? No if not, there are some items need to be check. 1. GC, but you say gc is not problem - I have verified GC multiple times and I don't see that to be an issue. 2. if you suspect network thread, ho

Re: High CPU Usage on Brokers

2020-01-08 Thread Ismael Juma
Has the behavior changed after an upgrade or has it been consistent since the start? Ismael On Thu, Jan 2, 2020 at 4:18 PM Navneeth Krishnan wrote: > Hi All, > > We have a kafka cluster with 12 nodes and we are pretty much seeing 90% > cpu usage on all the nodes. Here is all the information. Ne

Re: High CPU Usage on Brokers

2020-01-08 Thread Lisheng Wang
Hi Navneeth like the bug you said above, do you set sun.security.jgss.native = true? if not, there are some items need to be check. 1. GC, but you say gc is not problem 2. if you suspect network thread, how many thread did you set? 3. if you enable compression 4. did you change the value of bat

Re: High CPU Usage on Brokers

2020-01-07 Thread Navneeth Krishnan
Hi All, Any suggestions, we are running into this issue in production and any help would be greatly appreciated. Thanks On Mon, Jan 6, 2020 at 9:26 PM Navneeth Krishnan wrote: > Hi, > > Thanks for the response. We were using version 0.11 previously and all our > producers/consumers have been u

Re: High CPU Usage on Brokers

2020-01-06 Thread Navneeth Krishnan
Hi, Thanks for the response. We were using version 0.11 previously and all our producers/consumers have been upgraded to either 1.0 or to the latest 2.3. Is it normal for the network thread to consume more cpu? If you look at it, the network thread consumes 50% of the overall cpu. Regards On Mo

Re: High CPU Usage on Brokers

2020-01-06 Thread Thunder Stumpges
Not sure what version your producers/consumers are, or if you upgraded from a previous version that used to work, or what, but maybe you're hitting this? https://kafka.apache.org/23/documentation.html#upgrade_10_performance_impact On Mon, Jan 6, 2020 at 12:48 PM Navneeth Krishnan wrote: > Hi

Re: High CPU Usage on Brokers

2020-01-06 Thread Navneeth Krishnan
Hi All, Any idea on what can be done? Not sure if we are running into this below bug. https://issues.apache.org/jira/browse/KAFKA-7925 Thanks On Thu, Jan 2, 2020 at 4:18 PM Navneeth Krishnan wrote: > Hi All, > > We have a kafka cluster with 12 nodes and we are pretty much seeing 90% > cpu usa