Re: Meaure server time of Get/Scan - through RPC logging?

2012-12-08 Thread Wei Tan
Hmm... I appreciate it if anybody can point me how to measure server side Get/Scan time... Thanks! Best Regards, Wei From: Wei Tan/Watson/IBM To: hbase-u...@hadoop.apache.org, Date: 12/02/2012 12:21 PM Subject:Meaure server time of Get/Scan - through RPC logging? Hi I am u

Re: Heterogeneous cluster

2012-12-08 Thread Jean-Marc Spaggiari
Hi Mike, I totally agree with you. The balancer I have done is more a hack than a production version. I built my cluster by taking all the computers I found around me. From 1 core P4 to 8 cores CPU. I have 8 nodes + 3 ZK. They are all SO diffrent that "normal" load balancing was not very efficient

Re: Heterogeneous cluster

2012-12-08 Thread Robert Dyer
I of course can not speak for Jean-Marc, however my use case is not very corporate. It is a small cluster (9 nodes) and only 1 of those nodes is different (drastically different). And yes, I configured it so that node has a lot more map slots. However, the problem is HBase balances without regar

Re: Heterogeneous cluster

2012-12-08 Thread Michael Segel
Take what I say with a grain of kosher salt. (Its what they put on your drink glasses because the grains are bigger. ;-) I think what you are doing is cool hack, however in the bigger picture, you shouldn't have to do this with your load balancer. Also it doesn't matter if you think about ti.

Re: Bulk loading (and/or major compaction) causing OOM

2012-12-08 Thread Bryan Beaudreault
Merging is not an option for us, because we cannot afford to bring our cluster down. Also, we are not yet convinced that our cluster can handle such large regions due to all the OOM issues we are seeing when trying to bring new, bigger regions online. On Sat, Dec 8, 2012 at 3:42 PM, Marcos Ortiz

Re: Bulk loading (and/or major compaction) causing OOM

2012-12-08 Thread Marcos Ortiz
On 12/08/2012 11:50 AM, Bryan Beaudreault wrote: Thanks for the responses guys. Responses inline When you are doing the bulk load, are you pre-split your regions? What OS are you using and what version of Java? Yes, regions are pre-split. We calculated them using M/R before attempting to bu

Re: Heterogeneous cluster

2012-12-08 Thread Jean-Marc Spaggiari
Hi, It's not yet available anywhere. I will post it today or tomorrow, just the time to remove some hardcoding I did into it ;) It's a quick and dirty PerformanceBalancer. It's not a CPULoadBalencer. Anyway, I will give more details over the week-end, but there is absolutly nothing extraordinaire

Re: HBase Integration with Active Directory

2012-12-08 Thread Harsh J
Hi, An KDC can be made to trust an AD, which would solve your need. This https://ccp.cloudera.com/display/CDH4DOC/Integrating+Hadoop+Security+with+Active+Directory is one guide that details on how to set it up. HBase wraps very little logic over Hadoop's security providing classes, so proper Hado

Re: Heterogeneous cluster

2012-12-08 Thread Robert Dyer
I too am interested in this custom load balancer, as I was actually just starting to look into writing one that does the same thing for my heterogeneous cluster! Is this available somewhere? On Sat, Dec 8, 2012 at 9:17 AM, James Chang wrote: > By the way, I saw you mentioned that you > hav

Re: HBase Integration with Active Directory

2012-12-08 Thread anil gupta
Hi Harsh, Both of the approach you mentioned would be ok for us. We are aware that Hadoop can be integrated with Active Directory. But, i could not find any such reference for HBase. Do you have any idea about this? Any link or documentation on this would be really helpful. Thanks, Anil Gupta On

Re: Bulk loading (and/or major compaction) causing OOM

2012-12-08 Thread Bryan Beaudreault
Thanks for the responses guys. Responses inline > When you are doing the bulk load, are you pre-split your regions? > What OS are you using and what version of Java? Yes, regions are pre-split. We calculated them using M/R before attempting to bulk load the data. We've done this before with sm

Re: HBase Integration with Active Directory

2012-12-08 Thread Harsh J
Do you want to have just AD (via LDAP) based authentication (not sure what I'm talking of here, really), or kerberos based authentication but with an automatic binding to AD (via LDAP) for all the allowed/available users? On Tue, Dec 4, 2012 at 11:19 AM, anil gupta wrote: > Hi All, > > We would l

Re: Heterogeneous cluster

2012-12-08 Thread James Chang
Hi JM, I ever think the same issue, in my opinion, option 2 is perfer. By the way, I saw you mentioned that you have built a "LoadBalancer", could you kindly share some detailed info about it? Best Regards. James Chang Jean-Marc Spaggiari 於 2012年12月8日星期六寫道: > Hi, > > Here is the s