Thanks for the explanation Qian!
I think being able to balance empty regions is important and the preferred
result to me.
The best way so far is to manually 'balance' the regions if we need to add
pre-splits dynamically.
Jianshi
On Tue, Sep 23, 2014 at 11:35 AM, Qiang Tian wrote:
> Hello,
I mean I have no idea why the RegionServer has so many log about
establishing and closing session.
I don't think we have so many clients within every second. This happens
when I restart the server but not for all HBase clusters.
On Tue, Sep 23, 2014 at 11:57 AM, Qiang Tian wrote:
> sorry, what
You can use the hbase.hstore.time.to.purge.deletes config option.
You can set it globally or per Column Family.
This is the description in hbase-default.xml:
hbase.hstore.time.to.purge.deletes
0
The amount of time to delay purging of delete markers with
future timestamps. If
sorry, what reason do you mean?
I took a quick look at 0.94.21, it looks only client code paths call below
constructor, so this version looks immune to the problem..
public CatalogTracker(final Configuration conf) throws IOException {
this(null, conf, null);
}
On Mon, Sep 22, 2014 at 7:4
Cycling previous bits (w.r.t. adjusting weights for table skew):
http://search-hadoop.com/m/DHED4CWSqW1/snapshot+timeout+problem&subj=Re+snapshot+timeout+problem
Cheers
On Mon, Sep 22, 2014 at 8:35 PM, Qiang Tian wrote:
> Hello, I happened to got balancer related issues 2 months ago and looked
considering 80-20 rule(does it apply to hbase tables?), what about
specifying balance property(per-table vs non per-table) on a table
basis(default is non per-table), then those few large tables could be
balanced well with acceptable total balance cost?
On Tue, Sep 23, 2014 at 11:35 AM, Qiang Tia
Hello, I happened to got balancer related issues 2 months ago and looked at
that part, below is a summary:
1)by default, hbase balancer(StochasticLoadBalancer by default) does not
balance regions per table. i.e. all regions are considered as 1 table. so
if you have many tables, especially some tab
The HBase Team is pleased to announce the immediate release of HBase 0.99.0.
Download it from your favorite Apache mirror [1] or maven repository.
THIS RELEASE IS NOT INTENDED FOR PRODUCTION USE, and does not contain any
backwards or forwards compatibility guarantees (even within minor versions
0.
Hi folks,
We have a hbase table with 4 column families which stores log data.The
columns and the content stored on each of these column families are the
same. The reason for having multiple families is that we needed 4 retention
buckets for messages and were using the TTL feature of hbase to
Hi Demai,
I am talking about point-search, there is no clearly pattern in the key.
I assume range-search is something like product id from 001 ~ 999 or
something like that.
Thanks,
Bin
On Mon, Sep 22, 2014 at 12:13 PM, Demai Ni wrote:
> Bin,
>
> Are you talking about point-search or range-se
Bin,
Are you talking about point-search or range-search?
While I think HBase is very good at point-search(comparing to other big
data store), I have similar questions about range-search. To get a good
performance of range-search, I am afraid of the hot-spot issue during bulk
load phase, and loo
https://issues.apache.org/jira/browse/HBASE-11368 related to the original
issue too.
On Mon, Sep 22, 2014 at 10:18 AM, Ted Yu wrote:
> As you noted in the FIXME, there're some factors which should be tackled by
> balancer / assignment manager.
>
> Please continue digging up master log so that we
Could tombstone purges be independent of purging deleted cells and
KEEP_DELETED_CELLS setting? In my use case, I do not want to keep deleted
cells, but I do need to keep the tombstones around. Without the tombstones,
I'm not able to do incremental backups (custom, we do timerange raw scans
ourselve
As you noted in the FIXME, there're some factors which should be tackled by
balancer / assignment manager.
Please continue digging up master log so that we can find the cause for
balancer not fulfilling your goal.
Cheers
On Mon, Sep 22, 2014 at 10:09 AM, Jianshi Huang
wrote:
> Ok, I fixed this
Ok, I fixed this by manually reassign region servers to newly created ones.
def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo],
regionServers: Seq[ServerName]): Unit = {
val rand = new Random()
regions.foreach { r =>
val idx = rand.nextInt(regionServers.size)
Hi Bin,
100M rows 1 byte is about 100MB.
100M rows 1KB is about 100GB.
What is your record size and what is your SLA?
Do you expected 100GB to be transfered in few seconds?
How to you query your data? A single get? All of it?
You might want to give way more details of your usecases if you want
Hmm...any workaround? I only want to do this:
Rebalance the new regions *evenly* to all servers after manually adding
splits, so later bulk insertions won't cause contention.
P.S.
Looks like two of the region servers which had majority of the regions were
down during Major compaction... I guess i
Hi Ted,
I have not dive into the programming part yet... I am still at the POC and
pick up the right tool stage. Based on your experience, do you think the
get(List gets) will return the result from 100M level in an
interactive time? say a few seconds?
If that is the case, I will start working on
Yes, I have access to Master UI, however logs/*.log cannot be opened or
downloaded, must be some security restrictions in the proxy...
Jianshi
On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu wrote:
> Do you have access to Master UI ?
>
> :60010/logs/ would show you list of log files.
>
> The you can v
Do you have access to Master UI ?
:60010/logs/ would show you list of log files.
The you can view :60010/logs/hbase--master-XXX.log
Cheers
On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang
wrote:
> Ah... I don't have access to HMaster logs... I need to ask the admin.
>
> Jianshi
>
> On Mon, Sep
Ah... I don't have access to HMaster logs... I need to ask the admin.
Jianshi
On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu wrote:
> bq. assign per-table balancer class
>
> No that I know of.
> Can you pastebin master log involving output from balancer ?
>
> Cheers
>
> On Mon, Sep 22, 2014 at 8:29 A
bq. upload a list of product ids
Have you looked at the following API in HTable ?
public Result[] get(List gets) throws IOException {
Cheers
On Mon, Sep 22, 2014 at 8:14 AM, Bin Wang wrote:
> Hi there,
>
> I have a use case that I need to do bulk look up in a table of size 100
> million key
bq. assign per-table balancer class
No that I know of.
Can you pastebin master log involving output from balancer ?
Cheers
On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang
wrote:
> Hi Ted,
>
> I moved setBalancerRunning before balancer and run them twice. However I
> still got highly skewed regi
Hi Ted,
I moved setBalancerRunning before balancer and run them twice. However I
still got highly skewed region distribution.
I guess it's because of the StochasticLoadBalancer, can I assign per-table
balancer class in HBase?
Jianshi
On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu wrote:
> admin.set
Hi there,
I have a use case that I need to do bulk look up in a table of size 100
million key value pairs. Where key is the unique ID (product id), and the
value is inventory history (time series) for that particular part.
I want user upload a list of product ids, and I am wondering if HBase is
t
admin.setBalancerRunning() call should precede the call to admin.balancer().
You can inspect master log to see whether regions are being moved off the
heavily loaded server.
Cheers
On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang
wrote:
> Hi Ted and others,
>
> I did the following after adding s
Hi Upen,
I will recommand you to jump on the Bigtop mailing list for those
questions.
You will have to download bigtop (bit bigtop-hbase), configure the config
file and use make hbase-rpm. Let's continue the discussion on the bigtop
side.
JM
2014-09-22 8:51 GMT-04:00 Upender Nimbekar :
> Thank
Thanks Jean for the prompt reply.
So I guess I have to dowload bigtop-hbase project, make changes to config
to update to desired version, build to create RPM. Let me know if there are
any additonal steps involved ?
Thanks
Upen
On Mon, Sep 22, 2014 at 8:11 AM, Jean-Marc Spaggiari <
jean-m...@spagg
Hi Upen,
Take a look at BigTop http://bigtop.apache.org/
It will create the rpm (and even deb if you use the amazing debian
distribution)
You can also create hadoop packages, Phoenix, etc.
Just edit the config file to package the version you want.
JM
2014-09-22 8:06 GMT-04:00 Upender Nimbekar
Fellow Hbase users and developers,
We have a need to use Hbase RPM for a specific version 0.94.23.
Hbase.apache.org ONLY has TAR file to download. But I need RPM for this
specific version. I would like to avoid creating RPMs manually right from
the scratch and would keep it as a last resort if I do
@qiang I have read about this issue. Is it
https://issues.apache.org/jira/browse/HBASE-4495?
I look deep in to code and can't find the reason of this. Any suggestion is
welcome.
On Mon, Sep 22, 2014 at 6:22 PM, Qiang Tian wrote:
> it looks coming from
>
> http://grepcode.com/file/repo1.maven.or
it looks coming from
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.19/org/apache/hadoop/hbase/catalog/CatalogTracker.java#186
if no zookeeper watcher passed in, we will connect to zk...this logic
mostly used in client side(getCatalogTracker), so it looks some server s
Hi Ted and others,
I did the following after adding splits (without data) to my table, however
the region is still very imbalanced (one region server has 221 regions and
other 50 region servers have about 4~8 regions each).
admin.balancer()
admin.setBalancerRunning(true, true)
The ba
33 matches
Mail list logo
