> The number of Completed HH tasks is interesting. AFAIK a task is started when
> the node detects another in the cluster has returned. Were you doing some
> other restarts around the cluster ?
Not at all. The restarts seem to happen as normal operation.
>
> I don't want to divert from the GC
The number of Completed HH tasks is interesting. AFAIK a task is started when
the node detects another in the cluster has returned. Were you doing some other
restarts around the cluster ?
I don't want to divert from the GC issue, just wondering if something else is
going on as well. Like the n
>> What does the TPStats look like on the nodes under pressure ? And how many
>> nodes are delivering hints to the nodes when they restart?
$nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed
ReadStage 1 11992475
Requ
> I'm assuming the two nodes are the ones receiving the HH after they were
> down.
Adjacent, so yes.
>
> Are there a lot of hints collected while they are down ? you can check the
> HintedHandOffManager MBean in JConsole
There wasn't any downtime - that's something else that's weird.
>
>
Greetings,
> Doesn't really look abnormal to me for a heavy write load situation
> which is what "receiving hints" is.
I would agree with you but this raises some questions about write
performance. Plus I've only seen this kind of behavior recently and only
on 2 adjacent nodes. So I have good rea
> An if you have 10 nodes, do all of them happen to send hints to the two
> with GC?
The 2 nodes are adjacent in token range. They are replicating to each other.
Other nodes have no data to replicate so there's no proof one way or
another.
Best,
Gabriel
Greetings,
> Just out of curiosity is this on the receiver or sender side?
Looks like sender side, although the 2 nodes were replicating to each
other so it's hard to tell.
>
> I have been wondering a bit if the hint playback could need some
> adjustment.
> There is potentially quite big diffe
An if you have 10 nodes, do all of them happen to send hints to the two with
GC?
Terje
On Thu, May 12, 2011 at 6:10 PM, Terje Marthinussen wrote:
> Just out of curiosity is this on the receiver or sender side?
>
> I have been wondering a bit if the hint playback could need some
> adjustment.
>
Just out of curiosity is this on the receiver or sender side?
I have been wondering a bit if the hint playback could need some
adjustment.
There is potentially quite big differences on how much is sent per throttle
delay time depending on what your data looks like.
Early 0.7 releases also built u
Doesn't really look abnormal to me for a heavy write load situation
which is what "receiving hints" is.
On Wed, May 11, 2011 at 1:55 PM, Gabriel Tataranu wrote:
> Greetings,
>
> I'm experiencing some issues with 2 nodes (out of more than 10). Right
> after startup (Listening for thrift clients...
I'm assuming the two nodes are the ones receiving the HH after they were down.
Are there a lot of hints collected while they are down ? you can check the
HintedHandOffManager MBean in JConsole
What does the TPStats look like on the nodes under pressure ? And how many
nodes are delivering hints
Greetings,
I'm experiencing some issues with 2 nodes (out of more than 10). Right
after startup (Listening for thrift clients...) the nodes will create
objects at high rate using all available CPU cores:
INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed
leaving 2024909864 used; m
12 matches
Mail list logo
