Hi, C*1.2.2.
I have removed 4 nodes with "nodetool decommission". 2 of them have left with no issue, while the 2 others nodes remained "leaving" even after streaming their data. The only specific thing of these 2 nodes is that they had a lot of hints pending. Hints from a node that couldn't come back and that I removed earlier (because of the heavy load induced by Hinted Handoff while coming back, which induced a lot of latencies in our app. This node didn't manage to come back after 10 minutes, I removed it). So there I faced 3 bugs (or problems) : 1 - At first, one of my node came down 5 min and when it came back it get flooded by Hinted Handoff so hard that it could not handle the real time queries properly. I haven't find a way to prioritize app queries rather than Hinted Handoff. 2 - Nodes keep hints for a node that has been removed. 3 - Nodes with 500MB to 3GB hints stored for a removed node can't be decommissioned, they stuck after streaming their data. As solutions for this 3 issues I did the following: Solution to 1 - I removed this down node (nodetool removenode) Solution to 2 - Stop the node remove system hints Solution to 3 - Stop the node and removenode instead of decommission Now I have no more issue, yet I felt I had to report this. Maybe my experience can help users to get out of tricky situations and commiters to detect some issues, specially about hinted handoff. Alain
