Re: unexplained hinted handoff

2013-04-16 Thread Dane Miller
On Sun, Apr 14, 2013 at 11:28 AM, aaron morton wrote: >> If hints are being stored, doesn't that imply DOWN nodes, and why don't I >> see that in the logs? > > Hints are stored for two reasons. First if the node is down when the write > request starts, second if the node does not reply to the coo

Re: unexplained hinted handoff

2013-04-14 Thread aaron morton
>> Do slow reads trigger hint storage? No. But dropped read messages is often an indicator that the node is overwhelmed. >> If hints are being stored, doesn't that imply DOWN nodes, and why don't I >> see that in the logs? Hints are stored for two reasons. First if the node is down when the wri

Re: unexplained hinted handoff

2013-04-12 Thread Dane Miller
On Fri, Apr 12, 2013 at 1:12 PM, Dane Miller wrote: > I'm seeing hinted handoff kick in on all our nodes during periods of > high activity, but all the nodes seem to be up (according to the logs > and nodetool status). The pattern in the logs is something like this: > > 18:10:45 194 READ messages

unexplained hinted handoff

2013-04-12 Thread Dane Miller
I'm seeing hinted handoff kick in on all our nodes during periods of high activity, but all the nodes seem to be up (according to the logs and nodetool status). The pattern in the logs is something like this: 18:10:45 194 READ messages dropped in last 5000ms 18:11:10 Started hinted handoff for ho