Re: Very high read rate in a single RegionServer

On Tue, Jan 27, 2015 at 12:31 PM, Pun Intended wrote: > On Tue, Jan 27, 2015 at 12:58 PM, Stack wrote: > > > On Tue, Jan 27, 2015 at 9:34 AM, Pun Intended > > wrote: > > > The majority of the MR jobs are doing heavy reads/scans. Also doing > bulk-loading twice as day. The slow tasks are the one

Re: Very high read rate in a single RegionServer

On Tue, Jan 27, 2015 at 12:58 PM, Stack wrote: > On Tue, Jan 27, 2015 at 9:34 AM, Pun Intended > wrote: > > > Hi There, > > > > Thank you for your reply. The tasks are processing a lot of data. Each RS > > is hosting 15 regions. I have a total of ~4000 regions. > > > > 4000/15 = ~270 servers? >

Re: Very high read rate in a single RegionServer

On Tue, Jan 27, 2015 at 9:34 AM, Pun Intended wrote: > Hi There, > > Thank you for your reply. The tasks are processing a lot of data. Each RS > is hosting 15 regions. I have a total of ~4000 regions. > 4000/15 = ~270 servers? > * The biggest ones are ~10Gbs Snappy-compressed (I think that's

Re: Very high read rate in a single RegionServer

Hi There, Thank you for your reply. The tasks are processing a lot of data. Each RS is hosting 15 regions. I have a total of ~4000 regions. * The biggest ones are ~10Gbs Snappy-compressed (I think that's pretty big and these are the slow ones). There are ~1000 of these. * Then there are about 20

Re: Very high read rate in a single RegionServer

On Sat, Jan 24, 2015 at 5:15 PM, Pun Intended wrote: > Hello, > > I have noticed lately that my apps started running longer. You are processing more data now? > The longest > running tasks seem all to be requesting data from a single region server. > Has the rate at which you access hbase:m

Re: Very high read rate in a single RegionServer

On Sun, Jan 25, 2015 at 5:28 PM, Ted Yu wrote: > After the adjustment, hbase:meta would still be accessed for region > locations. But the request count should come down noticeably. > > Why would compacting meta change the rate at which it is requested? St.Ack > bq. Is there a way to inspect wh

Re: Very high read rate in a single RegionServer

Please try to determine how much percent of the reqs/sec metric is due to read. At this level, do your queries / jobs get slower ? Cheers On Sun, Jan 25, 2015 at 6:48 PM, Pun Intended wrote: > Sorry for the confusion. I should have used the same place to grab the > numbers from. > > In my ori

Re: Very high read rate in a single RegionServer

Sorry for the confusion. I should have used the same place to grab the numbers from. In my original email, I grabbed the reqs/sec metric from the Hbase UI directly. It was showing 1000 Reqs/sec. Right now, when I look at that, the number is 891 Reqs/sec, but it fluctuates. The 55K Reqs/sec in m

Re: Very high read rate in a single RegionServer

>From your first email: bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere Now the request count becomes 55K requests/sec ? Just want to clarify whether the request count increased or decreased when compaction is not in progress. Cheers On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended wrote: > Thanks! >

Re: Very high read rate in a single RegionServer

Thanks! I ran hbase hbck and all seems healthy. 0 inconsistencies detected. Status: OK Also the meta region shows to be 100% compacted, but I still see 55K requests/sec to that RegionServer: Region Name Num. Comp

Re: Very high read rate in a single RegionServer

After the adjustment, hbase:meta would still be accessed for region locations. But the request count should come down noticeably. bq. Is there a way to inspect whether I have corruptions in the META table? Please take a look at: http://hbase.apache.org/book.html#hbck.in.depth Cheers On Sun, Jan

Re: Very high read rate in a single RegionServer

Will do! Thanks you for your suggestion. I will keep an eye on the read rate to that RegionServer, but in case it doesn't drop, do you think there should be something else I need to look at? * Is there a way to inspect whether I have corruptions in the META table? * Any other possible misconfigura

Re: Very high read rate in a single RegionServer

Suggest you lower the frequency of compacting hbase:meta Move the second command to another script. Cheers > On Jan 25, 2015, at 4:46 PM, Pun Intended wrote: > > Hi Ted, > > I have a small script: major_compact_hbase.hbsh > > major_compact 'users' > major_compact 'hbase:meta' > exit > >

Re: Very high read rate in a single RegionServer

Hi Ted, I have a small script: major_compact_hbase.hbsh major_compact 'users' major_compact 'hbase:meta' exit and a cron calls it once a day with: /usr/bin/hbase shell major_compact_hbase.hbsh Thanks again! On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu wrote: > How do you trigger major compacti

Re: Very high read rate in a single RegionServer

How do you trigger major compaction manually ? Cheers > On Jan 25, 2015, at 3:40 PM, Pun Intended wrote: > > Thank you so much for your help! > You are right, the slow region server hosts hbase:meta!!! I am using Hbase > 0.96.1.1 > > I have major compactions disabled. I run major compaction

Re: Very high read rate in a single RegionServer

Thank you so much for your help! You are right, the slow region server hosts hbase:meta!!! I am using Hbase 0.96.1.1 I have major compactions disabled. I run major compaction manually once a day and when it dpes that besides requesting major compaction on my tables, it also requests a major_compa

Re: Very high read rate in a single RegionServer

I assume you have read http://hbase.apache.org/book.html#perf.reading What hbase release are you using ? Does this region server host hbase:meta (assuming 0.98 release is used) ? Cheers On Sat, Jan 24, 2015 at 5:15 PM, Pun Intended wrote: > Hello, > > I have noticed lately that my apps starte

Very high read rate in a single RegionServer

Hello, I have noticed lately that my apps started running longer. The longest running tasks seem all to be requesting data from a single region server. That region server read rate is very high in comparison to the read rate of all the other region servers (1000reqs/sec vs 4-5 reqs/sec elsewhere).