Hi Lars, Ours is a problem of I/O wait and network bandwidth increase around the same time....
Lars, Sorry to say this... our's is a production cluster and we ideally should never want a downtime... Also lars, we had very miserable experience while upgrading from 0.92 to 0.94... There was a never a mention of change in split policy in the release notes... and the policy was not ideal for our cluster and it took us atleast a week to figure out that.... Our cluster runs on commodity hardware with big regions (5-10gb)... Region sever mem is 10gb... 2TB SATA Hard disks (5400 - 7200 rpm)... Internal network bandwidth is 1 gig So please suggest us any work around with 0.94.1.... On Sun, Sep 7, 2014 at 8:42 AM, lars hofhansl <[email protected]> wrote: > Thinking about it again, if you ran into a HBASE-7336 you'd see high CPU > load, but *not* IOWAIT. > 0.94 is at 0.94.23, you should upgrade. A lot of fixes, improvements, and > performance enhancements went in since 0.94.4. > You can do a rolling upgrade straight to 0.94.23. > > With that out of the way, can you post a jstack of the processes that > experience high wait times? > > -- Lars > > ------------------------------ > *From:* kiran <[email protected]> > *To:* [email protected]; lars hofhansl <[email protected]> > *Sent:* Saturday, September 6, 2014 11:30 AM > *Subject:* Re: HBase - Performance issue > > Lars, > > We are facing a similar situation on the similar cluster configuration... > We are having high I/O wait percentages on some machines in our cluster... > We have short circuit reads enabled but still we are facing the similar > problem.. the cpu wait goes upto 50% also in some case while issuing scan > commands with multiple threads.. Is there a work around other than applying > the patch for 0.94.4 ?? > > Thanks > Kiran > > > On Thu, Apr 25, 2013 at 12:12 AM, lars hofhansl <[email protected]> wrote: > > You may have run into https://issues.apache.org/jira/browse/HBASE-7336 > (which is in 0.94.4) > (Although I had not observed this effect as much when short circuit reads > are enabled) > > > > ----- Original Message ----- > From: kzurek <[email protected]> > To: [email protected] > Cc: > Sent: Wednesday, April 24, 2013 3:12 AM > Subject: HBase - Performance issue > > The problem is that when I'm putting my data (multithreaded client, ~30MB/s > traffic outgoing) into the cluster the load is equally spread over all > RegionServer with 3.5% average CPU wait time (average CPU user: 51%). When > I've added similar, mutlithreaded client that Scans for, let say, 100 last > samples of randomly generated key from chosen time range, I'm getting high > CPU wait time (20% and up) on two (or more if there is higher number of > threads, default 10) random RegionServers. Therefore, machines that held > those RS are getting very hot - one of the consequences is that number of > store file is constantly increasing, up to the maximum limit. Rest of the > RS > are having 10-12% CPU wait time and everything seems to be OK (number of > store files varies so they are being compacted and not increasing over > time). Any ideas? Maybe I could prioritize writes over reads somehow? Is > it > possible? If so what would be the best way to that and where it should be > placed - on the client or cluster side)? > > Cluster specification: > HBase Version 0.94.2-cdh4.2.0 > Hadoop Version 2.0.0-cdh4.2.0 > There are 6xDataNodes (5xHDD for storing data), 1xMasterNodes > Other settings: > - Bloom filters (ROWCOL) set > - Short circuit turned on > - HDFS Block Size: 128MB > - Java Heap Size of Namenode/Secondary Namenode in Bytes: 8 GiB > - Java Heap Size of HBase RegionServer in Bytes: 12 GiB > - Java Heap Size of HBase Master in Bytes: 4 GiB > - Java Heap Size of DataNode in Bytes: 1 GiB (default) > Number of regions per RegionServer: 19 (total 114 regions on 6 RS) > Key design: <UUID><TIMESTAMP> -> UUID: 1-10M, TIMESTAMP: 1-N > Table design: 1 column family with 20 columns of 8 bytes > > Get client: > Multiple threads > Each thread have its own tables instance with their Scanner. > Each thread have its own range of UUIDs and randomly draws beginning of > time > range to build rowkey properly (see above). > Each time Scan requests same amount of rows, but with random rowkey. > > > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/HBase-Performance-issue-tp4042836.html > Sent from the HBase User mailing list archive at Nabble.com. > > > > > -- > Thank you > Kiran Sarvabhotla > > -----Even a correct decision is wrong when it is taken late > > > > -- Thank you Kiran Sarvabhotla -----Even a correct decision is wrong when it is taken late
