Hi Ted, Since HBase 0.98.8 we do better caching those calls (see HBASE-12219) can you upgrade to that version? Or if you are using 0.94 and you can't upgrade to HBase 1 or 0.98 there is a patch in that JIRA for 0.94 that might improve the situation.
cheers, esteban. -- Cloudera, Inc. On Thu, Mar 26, 2015 at 4:22 PM, Ted Tuttle <[email protected]> wrote: > We have no regions in transition. > > As far as who's requesting the table descriptors: We are calling > HTableIterface.getTableDescriptor frequently. We are going to cache the > result to avoid the unnecessary network calls. > > Thanks for the hint! > > From: lars hofhansl [mailto:[email protected]] > Sent: Thursday, March 26, 2015 3:43 PM > To: [email protected] > Cc: Development > Subject: Re: master consumes large amount of CPU for days > > Hi Ted, > > Yes, it is safe to bounce the HMaster without taking the region servers > down. > > Are there any regions in transition (would be shown on the master's page)? > > All of the threads involved with > - > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) > @bci=14, line=309 (Compiled frame) > ... > - > org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) > @bci=50, line=1932 (Compiled frame) > > are a bit suspect. Who's requesting all the table descriptors from the > master? > -- Lars > > ________________________________ > From: Ted Tuttle <[email protected]<mailto:[email protected]>> > To: "[email protected]<mailto:[email protected]>" < > [email protected]<mailto:[email protected]>> > Cc: Development <[email protected]<mailto: > [email protected]>> > Sent: Thursday, March 26, 2015 12:18 PM > Subject: master consumes large amount of CPU for days > > Hello- > > Our master process started consuming a large amount of CPU (75% of box) > several days back and hasn't stopped. I have 2 questions: > > 1) what is it doing (stack dump and log below) > 2) is it safe to restart the master without taking the > whole cluster down? > > Master stack dump: > > http://pastebin.com/G0iNNEpC > > Master log from last 15 mins or so: > > http://pastebin.com/WQNjhFGf > > Thanks, > Ted > >
