-axis is
the number of seconds from start of the experiment. Gaps in the throughput
denote the problems. I have looked in the logs and it seems that the culprit
is the HDFS:
snippets of the logs of RS1 and RS2 where I found errors around one of the
gaps:
http://pastebin.ca/2063454
I have included also snippets of logs to datanodes to where the RS errors
pointed.
stack-3 wrote:
>
> St.Ack
>
>
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31646631.html
Sent from the HBase User mailing list archive at Nabble.com.
On Mon, May 16, 2011 at 4:55 AM, Stan Barton wrote:
>> Sorry. How do you enable overcommitment of memory, or do you mean to
>> say that your processes add up to more than the RAM you have?
>>
>
> The memory overcommitment is needed because in order to let java still
> "allocate" the memory for ex
t;>
>> On the version 0.20.6 I have seen long pauses during the importing phase
>> and
>> also when querying. I was measuring the how many queries were processed
>> per
>> second and could see pauses in the throughput. The only culprit I could
>&
On Fri, May 13, 2011 at 7:44 AM, Stan Barton wrote:
> stack-3 wrote:
>>
>> On Thu, Apr 28, 2011 at 6:54 AM, Stan Barton wrote:
>> Are you swapping Stan? You are close to the edge with your RAM
>> allocations. What do you have swappyness set to? Is it default?
>>
>> Writing you don't need that
; one RS. Right now I am trying to replay the incident.
> >>
> >> I know that my scenario would require better machines, but those are
> what
> >> I
> >> have now and am before production running stress tests. In comparison
> >> with
> >> 0.20.6 t
the latest posted patch does not do checksuming and if your hardware
> does not have ECC, it could be a problem).
>
>
I have done some test using random access queries and multiversioned data
(10 to 50 different timestamps per data) and that the random access in
v0.20.6 is degrading linearly with the number of versions, in the case of
0.90, some slow down was recorded but in sub-linear speed. Still while using
the same amount of machines.
The reads were random, I pre-selected the rows from the whole collection.
The cache helped, I could see in the pattern the time it took to serve a
query's answer from disk and from cache.
Are you sure that you have suggested the right patch (hdfs-237)? It mentions
dfsadmin... And no the machines do not have ECC enabled ram.
stack-3 wrote:
>
>
>
> St.Ack
>
>
Stan
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31612028.html
Sent from the HBase User mailing list archive at Nabble.com.
On Thu, Apr 28, 2011 at 6:54 AM, Stan Barton wrote:
>
> Yes, these high limits are for the user running the hadoop/hbase processes.
>
> The systems are ran on a cluster of 7 machines (1 master, 6 slaves). One
> processor, two cores and 3.5GB of memory. I am using about 800MB for hadoop
> (version
nfiguration and
> we'll take a look at them.
>
> Sounds like the hang is in the client if you can still get to the
> cluster from a new shell
>
> As Mike says, tell us more about your context. How many regions on
> each server. What is your payload like?
>
> T
On Wed, Apr 27, 2011 at 2:30 AM, Stan Barton wrote:
>
> Hi,
>
> what means increase? I checked on the client machines and the nproc limit is
> around 26k, that seems to be as sufficient. The same limit applies on the db
> machines...
>
The nproc and ulimits are 26k for the user who is running the
average load 42.0
>>>> 2011-04-06 19:03:15,252 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
>>>> scanning meta region {server: 184.106.69.238:60020, regionname:
>>>> .META.,,1.1028785192, startKey: <>}
>>>> 2011-04-06 19:03:15,265 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
>>>> scan of 40 row(s) of meta region {server: 184.106.69.238:60020,
>>>> regionname: .META.,,1.1028785192, startKey: <>} complete
>>>> 2011-04-06 19:03:15,266 INFO
>>>> org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s)
>>>> scanned
>>>>
>>>>
>>>> The region server logs:
>>>> 2011-04-06 19:02:21,294 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>>>> 2011-04-06 19:02:21,295 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>>>> sequenceid=1
>>>>
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31485864.html
Sent from the HBase User mailing list archive at Nabble.com.
: <>}
>>> 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 40 row(s) of meta region {server:
>>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: <>}
>>> complete
>>> 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>>
>>>
>>> The region server logs:
>>> 2011-04-06 19:02:21,294 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>>> 2011-04-06 19:02:21,295 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>>> sequenceid=1
>>>
>>>
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31477194.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
On Tue, Apr 26, 2011 at 11:36 AM, Jeff Whiting wrote:
> Would it make sense to do some kind of sanity check on these various
> configuration parameters when a region or master server starts? It seems
> like there is a lot of them and when they aren't right, it can cause big
> problems. Just have
Would it make sense to do some kind of sanity check on these various configuration parameters when a
region or master server starts? It seems like there is a lot of them and when they aren't right, it
can cause big problems. Just have it check the configuration parameters and output a warning i
gt;> 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 40 row(s) of meta region {server:
>>> 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: <>}
>>> complete
>>> 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>>
>>>
>>> The region server logs:
>>> 2011-04-06 19:02:21,294 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>>> 2011-04-06 19:02:21,295 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>>> sequenceid=1
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31480444.html
Sent from the HBase User mailing list archive at Nabble.com.
:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>>
>>
>> The region server logs:
>> 2011-04-06 19:02:21,294 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Creating region
>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
>> 2011-04-06 19:02:21,295 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
>> sequenceid=1
>>
>>
>
>
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31477194.html
Sent from the HBase User mailing list archive at Nabble.com.
2:20 PM
Subject: Re: HTable.put hangs on bulk loading
That exception means you are running out of threads on that whole
machine. I wonder how you were able to get that... is hbase running on
that machine too? I'd love you see your configuration but what you
pasted is the hbase-default,
;
>
> ____
> From: Ajay Govindarajan
> To: "user@hbase.apache.org"
> Sent: Thursday, April 7, 2011 5:35 PM
> Subject: Re: HTable.put hangs on bulk loading
>
> Thanks for pointing this out. I have uploaded the server config at:
> ht
base.apache.org"
Sent: Thursday, April 7, 2011 5:35 PM
Subject: Re: HTable.put hangs on bulk loading
Thanks for pointing this out. I have uploaded the server config at:
http://pastebin.com/U41QZGiq
thanks
-ajay
From: Jean-Daniel Cryans
To: user@hbase.apach
Thanks for pointing this out. I have uploaded the server config at:
http://pastebin.com/U41QZGiq
thanks
-ajay
From: Jean-Daniel Cryans
To: user@hbase.apache.org
Sent: Thursday, April 7, 2011 10:29 AM
Subject: Re: HTable.put hangs on bulk loading
There&#
.238:60020, regionname: .META.,,1.1028785192, startKey: <>}
> complete
> 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
>
>
> The region server logs:
> 2011-04-06 19:02:21,294 DEBUG
> org.apache.hadoop.hbase.regions
Scanner: All
> 1 .META. region(s) scanned
>
>
> The region server logs:
> 2011-04-06 19:02:21,294 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Creating region TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
> 2011-04-06 19:02:21,295 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
> sequenceid=1
>
> --
> View this message in context:
> http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31338874.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
95 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
sequenceid=1
--
View this message in context:
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31338874.html
Sent from the HBase User mailing list archive at Nabble.com.
22 matches
Mail list logo
