Hi Stack,
The NPE is this:
10/12/18 15:39:07 WARN hdfs.StateChange: DIR* FSDirectory.unprotectedSetTimes:
failed to setTimes
/hbase/inrdb_ris_update_rrc00/fe5090c366e326cf2b123502e2d4bcce/data/1350525083587292896
because source does not exist
10/12/18 15:39:07 WARN hdfs.StateChange: DIR* FSDire
On Sun, Dec 19, 2010 at 1:23 AM, Friso van Vollenhoven
wrote:
> Right now, however, I am in the unpleasant situation that my NN won't come up
> anymore after a restart (throws NPE), so I need to get that fixed first
> (without formatting, because I am not very keen on running the 6 day job
> ag
That was a great thread Todd... it almost got somewhere. Looks like
you were owed a response by the hotspot-gc crew.
Friso, I wonder if u23 is better? There are a bunch of G1 fixes in
it: http://www.oracle.com/technetwork/java/javase/2col/6u23bugfixes-191074.html
St.Ack
On Sat, Dec 18, 2010 at
Swappiness is set to zero, but swap is not disabled all together, so when RAM
gets over utilized the boxes will start swapping.
I am running G1 with defaults (no max pause given) on JVM 1.6.0_21. When not
swapping, it shows pauses of around 10s for full GCs on a 16GB heap, which do
not happen v
I also had long GC pauses on G1 when I tried it last summer. Check out
this thread for the gory details:
http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2010-July/000653.html
-Todd
On Sat, Dec 18, 2010 at 10:37 PM, Stack wrote:
> Setting swap to zero is going to an extreme but changing it
Setting swap to zero is going to an extreme but changing it so its not
the default -- 60% IIRC -- could help.
I made HBASE-3376 for the NPE. I'll take a look into it.
Congrats on the 113second G1 pause. I thought G1 was to be the end of
such pauses? (Smile). Its good to hear that its staying
Hey Friso,
> The GC is G1, so it may look different from what you expect
> from a GC log. I know it is considered experimental, but I
> like the concept of it and think it's nice to gain some
> experience with it.
You are probably the first to use the G1 GC seriously with HBase. Would love to
he
Hi J-D,
I redid the job as before to once more verify. The real problem appears to be
something I had not noticed, because I never expected it. The machines started
swapping during the job. I did not expect that, because there is a total of
about 45GB heap allocated on a 64GB machine and nothin
If it wasn't GC it would be core dumps and whatnot, nothings free :)
I will reiterate what I said in my first answer, I'd like to see your
GC log since at this point I haven't seen direct evidence of GC
activity.
J-D
On Fri, Dec 17, 2010 at 1:27 AM, Friso van Vollenhoven
wrote:
> Hi J-D,
>
> Th
Hi J-D,
Thanks for your comments and clarification. I guess GC does blow (especially
when writing things like databases and filesystems).
Right now I will dive into GC tuning once more and probably lower the number of
reducers on the insert jobs.
Thanks,
Friso
On 16 dec 2010, at 19:00, Jea
Thanks for the confirmation of the order of the slow down.
IO and network are doing OK. Main problem appears to be GC, like J-D pointed
out.
Thanks,
Friso
On 16 dec 2010, at 18:26, tsuna wrote:
> On Wed, Dec 15, 2010 at 6:44 AM, Friso van Vollenhoven
> wrote:
>> On the master UI HBase sh
> I am not surprised by the fact that there is a performance hit, I just
> expected it to be less. I figured it to be somewhere between 2 and 3 times
> slower, not 5 times. So with my question I was basically looking for some
> measure of what to expect based on someone else's experience. Apart
On Wed, Dec 15, 2010 at 6:44 AM, Friso van Vollenhoven
wrote:
> On the master UI HBase shows doing between 10K and 50K requests per second
> with quite some drops to almost zero for some amount of time, while without
> WAL for the same job it easily reaches over 100K sustained.
Those numbers ar
Hi J-D,
Some more inline...
Hey Friso,
A few thoughts:
- Using the WAL will degrade your performance a lot and this is
expected. Remember, appending is the operation of sending a chunk of
data to the memory of 3 datanodes in a pipeline (compared to just
writing to memory when not using it). I'm
Hey Friso,
A few thoughts:
- Using the WAL will degrade your performance a lot and this is
expected. Remember, appending is the operation of sending a chunk of
data to the memory of 3 datanodes in a pipeline (compared to just
writing to memory when not using it). I'm not sure I quite understand
Hi,
I am experiencing some performance issues doing puts with WAL enabled. Without,
everything runs fine.
My workload is doing roughly 30 million reads (rows) and after each read do a
number of puts (update multiple indexes, basically). Total is about 165 million
puts. The work is done from a
16 matches
Mail list logo
