@Frank, regarding write amplification: 1. What is your flush size? default is 128 MB. You should increase your "hbase.hregion.memstore.flush.size" so that you dont run over the limit of store files. 2. Have a look at "hbase.regionserver.global.memstore.lowerLimit". 3. Your heap size is also too big. Maybe you also run into GC issues. Have you checked your GC logs? 4. IMO, writes getting blocks at 9 files might be very less for a big Region Server. So, you can also consider increasing that.
On Fri, Mar 18, 2016 at 10:22 AM, Frank Luo <[email protected]> wrote: > Ted, > > Thanks for sharing. I learned something today. > > But I guess it doesn't apply to my case. It is true that I only run a > client for a few hours in a day, but the data is not date based. > > -----Original Message----- > From: Ted Yu [mailto:[email protected]] > Sent: Friday, March 18, 2016 12:10 PM > To: [email protected] > Subject: Re: is it a good idea to disable tables not currently hot? > > Frank: > Can you take a look at the following to see if it may help with your use > case(s) ? > > HBASE-15181 A simple implementation of date based tiered compaction > > Cheers > > On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <[email protected]> wrote: > > > There are two reasons I am hesitating going that route. > > > > One is that most of tables are fairly small. Going to 10GB will force > > tables to shrink to some nodes but not evenly distributed around the > > cluster, hence discouraging parallelism. But I think I can manage this > > issue if the second is resolved. > > > > The second issue, which I have battled with for two years now, is that > > I am doing online puts, which occasionally triggers compacts when a > > region is heavily inserted, and whenever it happens, all subsequent > > read/write are all on hold and I can see time out error on the client > > side. A typical compact runs for 4 minutes now and I have to increase > > timeout on a number of places to accommodate that. So if I increase > > the size to 10 GB, will compact time double? > > > > -----Original Message----- > > From: Jean-Marc Spaggiari [mailto:[email protected]] > > Sent: Friday, March 18, 2016 11:34 AM > > To: user > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > So you can safely increase your maximum region size to 10GB, which > > will divide the number of regions by 2. When you will be on 1.1.2 you > > can also do online merge to reduce this number of regions. The might > help too. > > > > JMS > > > > 2016-03-18 12:32 GMT-04:00 Frank Luo <[email protected]>: > > > > > 0.98 on hdp 2.2 currently. > > > > > > Soon will be on hdp2.3.4, which has HBase 1.1.2. > > > > > > -----Original Message----- > > > From: Jean-Marc Spaggiari [mailto:[email protected]] > > > Sent: Friday, March 18, 2016 11:29 AM > > > To: user > > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > > > Hi Frank, > > > > > > It might be doable. > > > > > > What HBase version are you running? > > > > > > JMS > > > > > > 2016-03-18 12:25 GMT-04:00 Frank Luo <[email protected]>: > > > > > > > No one has experience disabling tables? > > > > > > > > -----Original Message----- > > > > From: Frank Luo [mailto:[email protected]] > > > > Sent: Thursday, March 17, 2016 4:51 PM > > > > To: [email protected] > > > > Subject: is it a good idea to disable tables not currently hot? > > > > > > > > We have a multi tenants environment and each client occupies x > > > > number of hbase regions. We currently have about 500 regions per > > > > region server and I understand the guideline is less than 200. So > > > > we need to reduce the region counts. Increasing region file size > > > > is no more an option because we are already at 5G and I don’t want > > > > to go > > higher. > > > > > > > > Due to our unique use cases, all clients are running for a few > > > > hours in a day, then being quiet for the rest of time. So I am > > > > thinking whether it is a good idea to disable all quiet tables and > > > > only enable them when they are ready to run. Does anyone have > > > > experience on > > that? > > > > > > > > One thing I worry about is the Balancer. I am pretty sure the > > > > balancer will be confused when regions come and go. And I cannot > > > > afford not to have it running in case of region server crashes and > > > > come back. So doesn’t anyone have good ideas how to handle it? > > > > > > > > I already doing compact myself so that is not an issue. > > > > > > > > Another related question, if a region is enabled but not active > > > > read/write, how much resources it takes in terms of region server? > > > > > > > > Thanks! > > > > > > > > Frank Luo > > > > > > > > > > Merkle was named a leader in Customer Insights Services Providers by > > > Forrester Research < > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing- > > > ag > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=ema > > > il footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > > Forrester Research report names 500friends, a Merkle Company, a > > > leader in customer Loyalty Solutions for Midsize Organizations< > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing- > > > ag > > > ency/awards-recognition/500friends-merkle-company-named?utm_source=e > > > ma ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > This email and any attachments transmitted with it are intended for > > > use by the intended recipient(s) only. If you have received this > > > email in error, please notify the sender immediately and then delete > > > it. If you are not the intended recipient, you must not keep, use, > > > disclose, copy or distribute this email without the author’s prior > permission. > > > We take precautions to minimize the risk of transmitting software > > > viruses, but we advise you to perform your own virus checks on any > > > attachment to this message. We cannot accept liability for any loss > > > or damage caused by software viruses. The information contained in > > > this communication may be confidential and may be subject to the > > attorney-client privilege. > > > > > Merkle was named a leader in Customer Insights Services Providers by > > Forrester Research < > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=email > > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > Forrester Research report names 500friends, a Merkle Company, a leader > > in customer Loyalty Solutions for Midsize Organizations< > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag > > ency/awards-recognition/500friends-merkle-company-named?utm_source=ema > > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > This email and any attachments transmitted with it are intended for > > use by the intended recipient(s) only. If you have received this email > > in error, please notify the sender immediately and then delete it. If > > you are not the intended recipient, you must not keep, use, disclose, > > copy or distribute this email without the author’s prior permission. > > We take precautions to minimize the risk of transmitting software > > viruses, but we advise you to perform your own virus checks on any > > attachment to this message. We cannot accept liability for any loss or > > damage caused by software viruses. The information contained in this > > communication may be confidential and may be subject to the > attorney-client privilege. > > > Merkle was named a leader in Customer Insights Services Providers by > Forrester Research > < > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > Forrester Research report names 500friends, a Merkle Company, a leader in > customer Loyalty Solutions for Midsize Organizations< > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > -- Thanks & Regards, Anil Gupta
