@Ryan
Hmmm. Interesting.
Lets take a step back for a second.
Which do you prefer: A push model or a pull/poll model?
That would kind of dictate the decisions you would make in terms of design.
To your point about moving away from ZK, it would mean putting those features
in to HBase directly
bq. it'd be nice to minimize dependency on ZK
+1
This is happening in hbase trunk already.
Cheers
On Fri, Sep 5, 2014 at 11:17 AM, Ryan Rawson wrote:
> I guess my thought is that it'd be nice to minimize dependency on ZK,
> and eventually remove it all together. It just adds too much
> deploy
I guess my thought is that it'd be nice to minimize dependency on ZK,
and eventually remove it all together. It just adds too much
deployment complexity, and code complexity -- about 1 lines of
code.
I do like the notion of HBase self-hosting it's own performance data,
it's what Oracle and ot
@Ted,
Yes, that’s the general idea or rather a specific use case for what I was
thinking.
So it would be a different mechanism to help manage the information.
I would think that it would result in faster access to the information.
This would be very important if one were to do some query opt
The HBase Team is pleased to announce the immediate release of HBase 0.94.23.
Download it from your favorite Apache mirror [1].
This release has also been pushed to Apache's maven repository.
All previous 0.92.x and 0.94.x releases can be upgraded to 0.94.23 via a
rolling upgrade without downtime
Hmmm.
I don’t know.
I’d rethink that position.
There are some interesting designs that would suggest that placing it in ZK
would offer much better performance.
And the amount of data wouldn’t really justify a separate table.
I’ll have to check out the JIRA link…
Just something to think
Yes. See the following method in HBaseAdmin:
public boolean balancer()
On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang
wrote:
> Thanks Ted!
>
> Didn't know I still need to run the 'balancer' command.
>
> Is there a way to do it programmatically?
>
> Jianshi
>
>
>
> On Sat, Sep 6, 2014 at 12:29
Thanks Ted!
Didn't know I still need to run the 'balancer' command.
Is there a way to do it programmatically?
Jianshi
On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu wrote:
> After splitting the region, you may need to run balancer to spread the new
> regions out.
>
> Cheers
>
>
> On Fri, Sep 5, 20
Hi Steven,
I did 1) and 2) and the error was during LoadIncrementalHFiles.
I can't do 3) because that CF is mostly used for mapreduce inputs, so a
continuous rowkey is preferred.
Jianshi
On Sat, Sep 6, 2014 at 12:29 AM, Magana-zook, Steven Alan <
maganazo...@llnl.gov> wrote:
> Jianshi,
>
> I
After splitting the region, you may need to run balancer to spread the new
regions out.
Cheers
On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang
wrote:
> Hi Shahab,
>
> I see, that seems to be the right way...
>
>
> On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus
> wrote:
>
> > Shahab
>
>
>
>
>
> -
Jianshi,
I have seen many solutions to importing this kind of data:
1. Pre-splitting regions (I did not try this)
2. Using a map reduce job to create HFiles instead of putting individual
rows into the database
(instructions here: http://hbase.apache.org/book/arch.bulk.load.html
3. Modifying the
Hi Shahab,
I see, that seems to be the right way...
On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus
wrote:
> Shahab
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
For monotonically increasing data: can you try to do pre-splitting of the
destination table? That can help in avoiding one region getting overloaded
at the time of bulkimport.
Regards,
Shahab
On Fri, Sep 5, 2014 at 12:14 PM, Jianshi Huang
wrote:
> Thanks Ted, I'll try to do a major compact.
>
Thanks Ted, I'll try to do a major compact.
Hi Steven,
Yes, most of my rows are hashed to make it randomly distributed, but one
column family has monotonically increasing rowkeys, and it's used for
recording sequence of events.
Do you have a solution how to bulk import this kind of data?
Jiansh
Hi Jianshi,
What are the field(s) in your row key? If your row key is monotonically
increasing then you will be sending all of your requests to one region
server. Even after the region splits, all new entries will keep punishing
one server (the region responsible for the split containing the new k
See HBASE-10882 where Victor's workaround was to do major compaction before
the bulkload.
Cheers
On Fri, Sep 5, 2014 at 8:15 AM, Jianshi Huang
wrote:
> I'm importing 2TB of generated HFiles to HBase and I constantly get the
> following errors:
>
> Caused by:
>
> org.apache.hadoop.hbase.ipc.Rem
If you files are on the same cluster, then this is not the issue. Do you
have the command line you run to launch your bulk load?
2014-09-05 11:54 GMT-04:00 Jianshi Huang :
> Hi JM,
>
> What do you mean by the 'destination cluster'? The files are in the same
> Hadoop/HDFS cluster where HBase is
Hi JM,
What do you mean by the 'destination cluster'? The files are in the same
Hadoop/HDFS cluster where HBase is running.
Do you mean do the bulk importing on HBase Master node?
Jianshi
On Fri, Sep 5, 2014 at 11:18 PM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:
> Hi Jianshi,
>
>
Hi Jianshi,
You might want to upload the file on the destination cluster first and then
re-run your bulk load from there. That way the transfer time will not be
taken into consideration for the timeout size the files will be local.
JM
2014-09-05 11:15 GMT-04:00 Jianshi Huang :
> I'm importing
I'm importing 2TB of generated HFiles to HBase and I constantly get the
following errors:
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException):
org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in
6 ms.
regionName=g
1.0 hasn't been released yet.
You can get source code from branch-1 of git repo.
Cheers
On Sep 4, 2014, at 11:52 PM, 牛兆捷 wrote:
> Hi Zhihong:
>
> Where can I get source code of release 1.0 ?
>
>
> 2014-09-04 11:04 GMT+08:00 Ted Yu :
>
>> Yes. See HBASE-5349 which is in the upcoming 1.0 r
Hi Esteban,
I would expect poor performance for awhile with the HDFS balancer running,
but I guess the mystery on my end was that the HDFS balancer logs claimed
that it is done in 5 minutes but the bad performance continues for 45. I
did try your suggestion though, to leave the HDFS balancer out,
22 matches
Mail list logo
