I want to add more information. we've already setup a completely new Kafka cluster with 0.8.2.2, and the old cluster is still there, but no consumer or producer, it's really wired that the CPU usage is still high.
also changed fetch.purgatory.purge.interval.requests and producer.purgatory.purge.interval.requests to 100, as I saw the high CPU usage with FetchRequestPurgatory.expire(), but no luck. On Fri, Mar 18, 2016 at 7:07 PM, li jinyu <disorder...@gmail.com> wrote: > hi all, > > we encountered problem with 0.8.1.1, as a lot of BadVersion error in log, > we believe it's caused by issue > https://issues.apache.org/jira/browse/KAFKA-1382, so we upgraded kafka to > 0.8.2.2. > > but we found that kafka is using high CPU. following is the result of > 'top' command: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 16781 root 20 0 9072m 874m 17m S 975.5 5.5 471:27.56 java > > I tried to use HRPOF to measure the CPU usage, got the result: > CPU SAMPLES BEGIN (total = 139166340) Fri Mar 18 16:30:28 2016 rank self > accum count trace method 1 37.63% 37.63% 52372145 325245 > sun.nio.ch.EPollArrayWrapper.epollWait 2 33.37% 71.00% 46441166 319805 > sun.nio.ch.EPollArrayWrapper.epollWait 3 8.49% 79.50% 11819885 302339 > java.net.PlainSocketImpl.socketAccept 4 4.25% 83.74% 5909221 302910 > java.net.PlainSocketImpl.socketAccept 5 4.23% 87.98% 5891650 310256 > sun.nio.ch.EPollArrayWrapper.epollWait 6 4.21% 92.19% 5859357 319905 > sun.nio.ch.EPollArrayWrapper.epollWait 7 4.16% 96.34% 5782672 336990 > java.net.SocketInputStream.socketRead0 8 1.99% 98.33% 2762683 320651 > sun.misc.Unsafe.park 9 0.27% 98.60% 375233 319447 sun.misc.Unsafe.park 10 > 0.25% 98.85% 351701 320536 sun.misc.Unsafe.park 11 0.24% 99.09% 329232 > 329274 sun.misc.Unsafe.park 12 0.08% 99.17% 109428 325405 > sun.nio.ch.EPollArrayWrapper.epollCtl 13 0.05% 99.22% 69762 329549 > sun.nio.ch.EPollArrayWrapper.interrupt 14 0.03% 99.25% 45197 328439 > kafka.network.RequestChannel$Request.updateRequestMetrics 15 0.03% 99.28% > 38496 328333 sun.nio.ch.FileDispatcherImpl.write0 16 0.02% 99.30% 33307 > 337909 java.lang.System.identityHashCode 17 0.02% 99.32% 31783 328763 > sun.nio.ch.NativeThread.current 18 0.02% 99.34% 30917 322537 > sun.misc.Unsafe.unpark 19 0.02% 99.37% 29943 325979 > sun.nio.ch.EPollArrayWrapper.epollWait 20 0.02% 99.39% 27788 328474 > sun.nio.ch.FileDispatcherImpl.write0 21 0.02% 99.40% 26448 328397 > sun.nio.ch.FileDispatcherImpl.write0 22 0.02% 99.42% 25177 329364 > scala.collection.mutable.HashTable$class.$init$ 23 0.02% 99.44% 24230 > 328786 java.lang.Thread.currentThread 24 0.02% 99.46% 22683 328795 > java.lang.Thread.currentThread 25 0.01% 99.47% 16264 328092 > sun.nio.ch.IOUtil.drain > > as most of time, epollwait shouldn't be so expensive, it should be the > problem of way of statistic. > then I tried YourKit, found that most heavy method is in > FetchRequestPurgatory.expire(), I uploaded the screenshot of YourKit: > http://imgur.com/2tXjhBI > > does anyone have same problem? > > -- > Don't schedule every day, make them disorder. > -- Don't schedule every day, make them disorder.
