Re: Restart node => hinted handoff flood

2013-07-04 Thread MichaƂ Michalski
My blind guess is: https://issues.apache.org/jira/browse/CASSANDRA-5179 In our case the only sensible solution was to pause hints delivery and disable storing them (both done with a nodetool: pausehandoff and disablehandoff). Once they TTL'd (3 hours by default I believe?) I turned HH on again

Re: Restart node => hinted handoff flood

2013-07-04 Thread Alain RODRIGUEZ
The point is that there is no way, afaik, to limit the speed of these Hinted Handoff since it's not a stream like repair or bootstrap, no way either to keep the node out of the ring during the time it is receiving hints since hints and "normal" traffic both go through gossip protocol on port 7000.

Restart node => hinted handoff flood

2013-07-04 Thread Alain RODRIGUEZ
Hi, Using C*1.2.2 12 EC2 xLarge cluster. When I restart a node, if it spend a few minutes down, when I bring it up, all the cpu are blocked at 100%, even once compactions are disabled, inducing a very big and intolerable latency in my app. I suspect Hinted Handoff to be the cause of this. disabli