>> Do slow reads trigger hint storage? No. But dropped read messages is often an indicator that the node is overwhelmed.
>> If hints are being stored, doesn't that imply DOWN nodes, and why don't I >> see that in the logs? Hints are stored for two reasons. First if the node is down when the write request starts, second if the node does not reply to the coordinator before rpc_timeout. If you are not seeing dropped write messages it may indicate network issues between the nodes. >> I'm seeing hinted handoff kick in on all our nodes during periods of >> high activity, Are you seeing log messages about hints been sent to nodes? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 13/04/2013, at 8:23 AM, Dane Miller <d...@optimalsocial.com> wrote: > On Fri, Apr 12, 2013 at 1:12 PM, Dane Miller <d...@optimalsocial.com> wrote: >> I'm seeing hinted handoff kick in on all our nodes during periods of >> high activity, but all the nodes seem to be up (according to the logs >> and nodetool status). The pattern in the logs is something like this: >> >> 18:10:45 194 READ messages dropped in last 5000ms >> 18:11:10 Started hinted handoff for host: >> 7668c813-41a9-4d42-b362-5420528fefa0 with IP: /10.... >> 18:11:11 Finished hinted handoff of 13 rows to endpoint /10.... >> >> This happens on all the nodes every 10 min, and with a different >> endpoint each time. tpstats shows thousands of dropped reads, but no >> other types of messages are dropped. >> >> Do slow reads trigger hint storage? If hints are being stored, >> doesn't that imply DOWN nodes, and why don't I see that in the logs? > > Sorry, meant to add: Cassandra 1.2.3, Ubuntu 12.04 x64
