Schema migration in Hbase

2014-10-15 Thread Vimal Jain
Hi, I have a hbase table( say test1) with 3 cfs ( a,b,c) and i have bunch of cqs in each of these cf. I also have one more table ( say test2) with some cfs on same cluster. My requirement is to move data from test1 to test2 through my custom logic ( as in which cf:cq from test1 will go to which cf:

Re: custom filter on hbase 0.96

2014-10-15 Thread Nishanth S
Thanks Ted .I will take a look. -Nishanth On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu wrote: > Nishanth: > Good question. > > As a general coding guide, writing unit test is always a good start. Using > Matt's case as an example, take a look at TestPrefixFilter. > > There're various unit tests for

Re: custom filter on hbase 0.96

2014-10-15 Thread Ted Yu
Matt: I was answering Nishanth's question - testing custom filter on client side. Cheers On Wed, Oct 15, 2014 at 2:54 PM, Matt K wrote: > Ted, sorry, I have no idea what you mean. Could you please elaborate? > > On Wednesday, October 15, 2014, Ted Yu wrote: > > > Nishanth: > > Good question. >

Re: custom filter on hbase 0.96

2014-10-15 Thread Matt K
Ted, sorry, I have no idea what you mean. Could you please elaborate? On Wednesday, October 15, 2014, Ted Yu wrote: > Nishanth: > Good question. > > As a general coding guide, writing unit test is always a good start. Using > Matt's case as an example, take a look at TestPrefixFilter. > > There'

Re: custom filter on hbase 0.96

2014-10-15 Thread Ted Yu
Nishanth: Good question. As a general coding guide, writing unit test is always a good start. Using Matt's case as an example, take a look at TestPrefixFilter. There're various unit tests for Filters in hbase code. Cheers On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S wrote: > Hi Ted , > Since

Re: custom filter on hbase 0.96

2014-10-15 Thread Nishanth S
Hi Ted , Since I am also working on similar thing is there a way we can first test the filter on client side?.You know what I mean without disrupting others who are using the same cluster for other work? Thanks, Nishanth On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu wrote: > bq. Or create a new fi

Re: custom filter on hbase 0.96

2014-10-15 Thread Ted Yu
bq. Or create a new file, compile it into ... You should go with the above approach. On Wed, Oct 15, 2014 at 2:08 PM, Matt K wrote: > Hi all, > > I'm trying to get a custom filter to work on HBase 0.96. After some > searching, I found that starting from 0.96, the implementer is required to > im

custom filter on hbase 0.96

2014-10-15 Thread Matt K
Hi all, I'm trying to get a custom filter to work on HBase 0.96. After some searching, I found that starting from 0.96, the implementer is required to implement "toByteArray" and "parseFrom" methods, using Protocol Buffers. But I'm having trouble with the "how". The "proto" file for the existing

RE: HBase read performance

2014-10-15 Thread Khaled Elmeleegy
Done. It's HBASE-12270 Please let me know if you have any comments/questions. > Date: Mon, 13 Oct 2014 22:42:37 -0500 > Subject: RE: HBase read performance > From: bus...@cloudera.com > To: user@hbase.apache.org > > Yes, please file a JIRA and attach the t

Re: How to add HBase dependencies and conf with spark-submit?

2014-10-15 Thread Soumitra Kumar
I am writing to HBase, following are my options: export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar spark-submit \ --jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase

Re: How to add HBase dependencies and conf with spark-submit?

2014-10-15 Thread Fengyun RAO
+user@hbase 2014-10-15 20:48 GMT+08:00 Fengyun RAO : > We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an > HBase table in Spark program. > > I notice there are: > spark.driver.extraClassPath > spark.executor.extraClassPathproperties to manage extra ClassPath, over > even

Re: Is there a TableInputFormat implementation that supports multiple splits for each region

2014-10-15 Thread Azuryy Yu
I don't think we have it currently. On Wed, Oct 15, 2014 at 6:21 PM, Jianshi Huang wrote: > It seems each region is a split in current TableInputFormat. We have large > regions and it's suboptimal. > > Is there a TableInputFormat implementation that supports multiple splits > for each region? >

Is there a TableInputFormat implementation that supports multiple splits for each region

2014-10-15 Thread Jianshi Huang
It seems each region is a split in current TableInputFormat. We have large regions and it's suboptimal. Is there a TableInputFormat implementation that supports multiple splits for each region? Thanks, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.co

Re: Slow Scan can loop forever

2014-10-15 Thread Qiang Tian
forgot below code: this.scan.setStartRow(this.lastResult.getRow()); we reset scanner to original position, so if we want to retry for OutOfOrderScannerNextException, perhaps one retry would be sufficient(since the exception already indicates do not retry...) filed HBASE-12266 On Wed, Oct 15, 2

Re: Slow Scan can loop forever

2014-10-15 Thread Qiang Tian
James, thanks for the details, an interesting case. It looks to me the OutOfOrderScannerNextException thrown by RS is correct -- note the internal cursor in scanner has moved on, even if the client side retries, client will not get the correct data(will lose data that the scanner already scanned)