> The number of Completed HH tasks is interesting. AFAIK a task is started when
> the node detects another in the cluster has returned. Were you doing some
> other restarts around the cluster ?
Not at all. The restarts seem to happen as normal operation.
>
> I don't want to divert from the GC
>> What does the TPStats look like on the nodes under pressure ? And how many
>> nodes are delivering hints to the nodes when they restart?
$nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed
ReadStage 1 11992475
Requ
> I'm assuming the two nodes are the ones receiving the HH after they were
> down.
Adjacent, so yes.
>
> Are there a lot of hints collected while they are down ? you can check the
> HintedHandOffManager MBean in JConsole
There wasn't any downtime - that's something else that's weird.
>
>
Greetings,
> Doesn't really look abnormal to me for a heavy write load situation
> which is what "receiving hints" is.
I would agree with you but this raises some questions about write
performance. Plus I've only seen this kind of behavior recently and only
on 2 adjacent nodes. So I have good rea
> An if you have 10 nodes, do all of them happen to send hints to the two
> with GC?
The 2 nodes are adjacent in token range. They are replicating to each other.
Other nodes have no data to replicate so there's no proof one way or
another.
Best,
Gabriel
Greetings,
> Just out of curiosity is this on the receiver or sender side?
Looks like sender side, although the 2 nodes were replicating to each
other so it's hard to tell.
>
> I have been wondering a bit if the hint playback could need some
> adjustment.
> There is potentially quite big diffe
Greetings,
I'm experiencing some issues with 2 nodes (out of more than 10). Right
after startup (Listening for thrift clients...) the nodes will create
objects at high rate using all available CPU cores:
INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed
leaving 2024909864 used; m
