Re: System hints compaction stuck

2013-08-07 Thread David McNelis
Fwiw, similar to another issue of stuck compaction that was on the list several days ago, if I cleared out the hints, either by removing files while node was down, or running a scrub on system.hints during node startup, I was able to get these compactions cleared, an the nodes are starting to get c

Re: System hints compaction stuck

2013-08-07 Thread Nate McCall
Is there anything else on the network that could be attempting to connect to 9160? That is the exact error you would get when someone initiates a connection and sends a null byte. You can reproduce it thusly: echo -n 'm' | nc localhost 9160 On Wed, Aug 7, 2013 at 11:11 AM, David McNelis wrote:

Re: System hints compaction stuck

2013-08-07 Thread David McNelis
Nate, We had a node that was flaking on us last week and had a lot of handoffs fail to that node. We ended up decommissioning that node entirely. I can't find the actual error we were getting at the time (logs have been rotated out), but currently we're not seeing any errors there. We haven't h

Re: System hints compaction stuck

2013-08-07 Thread Nate McCall
Thrift and ClientState are both unrelated to hints. What do you see in the logs after "Started hinted handoff for host:..." from HintedHandoffManager? It should either have an error message or something along the lines of "Finished hinted handoff of:..." Where there any schema updates that prece