I guess apache mailing list does not support attachments… Maybe you can paste it some where and send the link?
Jiangjie (Becket) Qin From: "pundlik.anuja" <pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com>> Reply-To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" <users@kafka.apache.org<mailto:users@kafka.apache.org>> Date: Monday, June 8, 2015 at 1:53 PM To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" <users@kafka.apache.org<mailto:users@kafka.apache.org>> Subject: Re: High CPU usage for idle kafka server Hi Jiangjie, Resending profiling screenshot for High CPU usage on Idle kafka broker. Appreciate your help. Thanks, Anuja On Mon, Jun 8, 2015 at 1:45 PM, pundlik.anuja <pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com>> wrote: Yes, partly it seems to be due to log cleaner thread. I changed log cleaning settings for each broker (server*.properties) file. # The minimum age of a log file to be eligible for deletion log.retention.hours=1 # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining # segments don't drop below log.retention.bytes. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=536870912 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms<http://log.retention.check.interval.ms>=10000 (****** This was 1 ms before) # By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires. # If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction. log.cleaner.enable=false (***** this was true before) Question: Per my understanding, setting log.cleaner.enable=false will cause deletion of logs instead of compaction. Is that correct? Or will it cause the logs to keep on accumulating and filling up the disk space? THanks, Anuja On Mon, Jun 8, 2015 at 1:24 PM, Jay Kreps <jay.kr...@gmail.com<mailto:jay.kr...@gmail.com>> wrote: Could it also be that the log cleaner is running? This will definitely use some CPU while the cleaning is occurring (it would attempt to use one cpu per log cleaner thread you configure). -Jay On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <j...@linkedin.com.invalid<mailto:j...@linkedin.com.invalid>> wrote: > It seems the attachments are lost. But high CPU for ReplicaRetcherThread > might be related to KAFKA-1461. Can you try to apply that patch and see if > it solves problem? > > Thanks, > > Jiangjie (Becket) Qin > > From: "pundlik.anuja" > <pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com><mailto: > pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com>>> > Reply-To: > "users@kafka.apache.org<mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>" > < > users@kafka.apache.org<mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>> > Date: Monday, June 8, 2015 at 12:20 PM > To: > "users@kafka.apache.org<mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>" > < > users@kafka.apache.org<mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>> > Subject: Re: High CPU usage for idle kafka server > > Seems to be LogCleaner and FetchRequest threads. > Attached is the profiling screenshot > > On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin > <j...@linkedin.com.invalid<mailto:j...@linkedin.com.invalid> > <mailto:j...@linkedin.com.invalid<mailto:j...@linkedin.com.invalid>>> wrote: > Has this to do with KAFKA-1461? > Can you see which thread is taking a lot of cpu? Some jconsole plugin can > get that information. > > Jiangjie (Becket) Qin > > On 6/5/15, 2:57 PM, "pundlik.anuja" > <pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com><mailto: > pundlik.an...@gmail.com<mailto:pundlik.an...@gmail.com>>> wrote: > > >Hi Jay, > > > >Good to hear from you. I met you at the kafka meetup at linkedin. > > > >- No, I am running kafka_2.11-0.8.2.1 > > > > > >Are there any logs/ any info that I can provide that will help you > >understand what could be the issue? > > > >Thanks, > >Anuja > > > >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps > ><jay.kr...@gmail.com<mailto:jay.kr...@gmail.com><mailto: > jay.kr...@gmail.com<mailto:jay.kr...@gmail.com>>> wrote: > > > >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are > >>running > >> that pre-release version is there? > >> > >> -Jay > >> > >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) < > >> apund...@cisco.com<mailto:apund...@cisco.com><mailto:apund...@cisco.com<mailto:apund...@cisco.com>> > >> > wrote: > >> > >> > Hi, > >> > > >> > I am using Kafka 0.8.2.1. > >> > We have 1 zookeeper, 3 kafka brokers. > >> > We have 9 topics, out of which 1 topic has 18 partitions, while > >>another > >> > has 12 partitions. All other topics have 1 partition each. > >> > > >> > We see that idle kafka brokers (not carrying any message) are using > >>more > >> > than 50% of CPU. See top output below. > >> > > >> > Is this a known issue? > >> > > >> > > >> > Thanks > >> > > >> > > >> > > >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92 > >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie > >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si, > >> > 0.4%st > >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k > >>buffers > >> > Swap: 0k total, 0k used, 0k free, 13382460k > >>cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java > >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java > >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java > >> > > >> > > >
