Re: hbasecon date at the website

2012-05-21 Thread Dani Rayan
I kept my word. Flying for HBasecon from LA... See you St.Ack :) On Thu, Feb 9, 2012 at 1:03 PM, Dani Rayan wrote: > Sure! Who wouldn't fly in to meet the gurus and have some HBase on the > rocks ;) > I'll let you know once I book the tickets and will get some cookies too! > > > -- Thanks, > -Da

Re: hbase insert performance test (from hbasemaster and regionservers)

2012-05-21 Thread Michael Segel
I would love to, but day job is still keeping me busy. :-) I think you can google up the other threads on this... Andrew wrote up a bit more. Sorry, -Mike On May 21, 2012, at 5:44 PM, Tom Brown wrote: > Micheal, > > This is good info. I wish you'd post what the "more" is, though. > > --Tom

Re: hbase insert performance test (from hbasemaster and regionservers)

2012-05-21 Thread Tom Brown
Micheal, This is good info. I wish you'd post what the "more" is, though. --Tom On Mon, May 21, 2012 at 4:30 PM, Michael Segel wrote: > Hi, > > Seems we just had someone talk about this just the other day... > > 1) 8GB of memory isn't enough to run both M/R and HBase. > Ok, yes you can run it,

Re: hbase insert performance test (from hbasemaster and regionservers)

2012-05-21 Thread Michael Segel
Hi, Seems we just had someone talk about this just the other day... 1) 8GB of memory isn't enough to run both M/R and HBase. Ok, yes you can run it, however don't expect it to perform well. 2) You never want a user to run their own code from the cluster itself. Use an *edge* node. There's mo

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Wouter Bolsterlee
Todd DeLuca schreef op ma 21-05-2012 om 15:32 [-0400]: > Please check your github pull requests for a fix for the thrift > dependency. Thanks, I've merged https://github.com/wbolster/happybase/pull/1 — Wouter signature.asc Description: This is a digitally signed message part

Re: Can we store a HBase Result object using Put

2012-05-21 Thread Jean-Daniel Cryans
How exactly are you building the Put? It doesn't have a constructor that can take byte[] and figure out how it should use it, it only takes a row key (meaning that if you do new Put(Result.getBytes().get()), you're passing the whole thing as a row key which is wrong). In the HBase code we do the R

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Todd DeLuca
Hi Wouter, Please check your github pull requests for a fix for the thrift dependency. In short, setup.py was dying when importing your package, before it had a chance to install the missing dependency. Now I need to go fix this in my stuff. :-) Cheers, Todd On Mon, May 21, 2012 at 1:51 PM, W

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Stack
On Mon, May 21, 2012 at 10:41 AM, Wouter Bolsterlee wrote: > Luke Lu schreef op ma 21-05-2012 om 10:38 [-0700]: >> Looks like it use the legacy thrift interface. > > Yes, it does. I might port it to Thrift 2 later, as mentioned in the > docs: http://readthedocs.org/docs/happybase/en/latest/todo.ht

Can we store a HBase Result object using Put

2012-05-21 Thread Shahsikant Jain
Hi, I am trying to store a result that I got out of some scan into another Hbase table so that I can read it back via Get and reconstruct. This is what I am doing 1. Result.getBytes().get() -- Get the byte[] and do a Put in HBase 2. Then do a get and read the bytes[] as new Result(new Immuta

hbase insert performance test (from hbasemaster and regionservers)

2012-05-21 Thread Faruk Berksöz
Dear All, we have 4 node in our cluster (1nn+dn,3 dn). Hadoop dist. is cdh3u3. Every node has 2 tb disk , 8 gb memory. We are trying some insert performance test on hbase. I have tried to insert 250.000 records from hbase master (without thread), that takes 5-7 sec. But when I try ro insert from

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Wouter Bolsterlee
Stack schreef op ma 21-05-2012 om 08:55 [-0700]: > Thanks for doing this. It looks great (Docs even!) You're welcome. And yes, it has documentation, and also unit tests with 100% code line coverage. :) — Wouter signature.asc Description: This is a digitally signed message part

Mahout's Text Similarity using HBase

2012-05-21 Thread Junaid Surve
Hello In my Project we are trying to calculate the Text Similarity of a set of documents for which I am facing 2 issues. 1. I do not want to recalculate the Term Frequency of the documents I have previously calculated. e.g. I have 10 docs and I have calculated the Term Frequency and

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Wouter Bolsterlee
Marcos Ortiz schreef op ma 21-05-2012 om 12:01 [-0400]: > +1 for this library, good design, it provides a clean and quick way to > interact with HBase using Python. > Can you upload it to PIP? Hi Marcos, I assume you meant PyPI, which pip uses. HappyBase is already available from PyPI: http://p

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Wouter Bolsterlee
Luke Lu schreef op ma 21-05-2012 om 10:38 [-0700]: > Looks like it use the legacy thrift interface. Yes, it does. I might port it to Thrift 2 later, as mentioned in the docs: http://readthedocs.org/docs/happybase/en/latest/todo.html > Stack, do you know > anybody using the new thrift2 interface?

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Luke Lu
Looks like it use the legacy thrift interface. Stack, do you know anybody using the new thrift2 interface? On Mon, May 21, 2012 at 8:55 AM, Stack wrote: > On Sun, May 20, 2012 at 3:14 PM, Wouter Bolsterlee wrote: >> Hi all, >> >> I'm happy to announce HappyBase, a developer-friendly Python libra

Re: Yet Another LeaseException :-(

2012-05-21 Thread Ted Yu
Thanks for the analysis. It shouldn't be difficult to verify your hypothesis. In the following code: } catch (Throwable t) { t = translateException(t); exceptions.add(t); You can add a log to show the type of t along with information about callable. When LeaseException

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Marcos Ortiz
+1 for this library, good design, it provides a clean and quick way to interact with HBase using Python. Can you upload it to PIP? Regards On 05/21/2012 03:25 AM, Leon Mergen wrote: On Mon, May 21, 2012 at 12:14 AM, Wouter Bolsterlee wrote: Hi all, I'm happy to announce HappyBase, a develop

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Stack
On Sun, May 20, 2012 at 3:14 PM, Wouter Bolsterlee wrote: > Hi all, > > I'm happy to announce HappyBase, a developer-friendly Python library to > interact with Apache HBase. > Thanks for doing this. It looks great (Docs even!) St.Ack

Re: key design

2012-05-21 Thread Lars George
Hi Mete, OpenTSDB uses the "natural" availability of the metrics ID to bucket the metrics by type. After that it relies on scanner batching, and block loads. For your uses case you could bin by time frames, say for example hash the start of each hour into an MD5 and concatenate it with the actu

Re: key design

2012-05-21 Thread Ian Varley
Mete, Why separate tables per log type? Why not a single table with the key: That's roughly the approach used by OpenTSDB (with "metric id" instead of "log type", but same idea). OpenTSDB goes further by "bucketing" values into rows using a base timestamp in the row key and offset timestamps

key design

2012-05-21 Thread mete
Hello folks, i am trying to come up with a nice key design for storing logs in the company. I am planning to index them and store row key in the index for random reads. I need to balance the writes equally between the R.S. and i could not understand how opentsdb does that with prefixing the metr

Yet Another LeaseException :-(

2012-05-21 Thread Igal Shilman
Hi, We've noticed in our production cluster (0.90.4-cdh3u3) that from time to time some of our map tasks fail due to a LeaseException thrown while scanning. We have "hbase.regionserver.lease.period", and "hbase.rpc.timeout" both set to 5 minutes. Whats strange about this, is the sequences of eve

Re: Garbage collection issues

2012-05-21 Thread Simon Kelly
Great, thanks very much for the help. I'm going to see if I can get more memory into the servers and will also experiment with XX:ParallelGCThreads. We already have XX:CMSInitiatingOccupancyFraction=70 in the config. Uday, what do you mean by "a fixed size record"? Do you mean the record that is b

Re: Why ScannerTimeoutException is not handled in TableInputFormat?

2012-05-21 Thread Jan Lukavský
Hi Vrushali, the issue is https://issues.apache.org/jira/browse/HBASE-5757. There are two patches, one is applicable against trunk, the other simple simple version should be applicable against hbase-0.90.5. It should fix your problem. Jan On 20.5.2012 23:26, Vrushali C wrote: Hi Jan I am

Re: Announcing HappyBase, a developer-friendly Python library to interact with HBase

2012-05-21 Thread Leon Mergen
On Mon, May 21, 2012 at 12:14 AM, Wouter Bolsterlee wrote: > Hi all, > > I'm happy to announce HappyBase, a developer-friendly Python library to > interact with Apache HBase. > > HappyBase is designed for for use in standard HBase setups, and offers > application developers a Pythonic API to inte