Hi all,
I am seeing a VERY large HintsColumnFamily (40+ GB) on one of my nodes (I
have 2 DC with 3 nodes each with 2 RF). Nodetool ring as a result reports load
as being way higher for the one node (the delta being the size of the
HintsColumnFamily). This behavior seems to occur if I do a large amount of
data loading using that node as the coordinator node. I found a post related
to this
(http://mail-archives.apache.org/mod_mbox/cassandra-user/201203.mbox/%3c376cec01195c894cb9f8a3c274029a96b471d...@fish-ex2k10-01.azaleos.net%3E)
but wanted to see if there were better ways to handle it then the reset
suggested as it seems somewhat risky. Nodetool netstats never seems to show
any streaming data. With past nodes it seemed like the node eventually fixed
itself. Note that I have the OOTB gc_grace_seconds so perhaps I just need to
wait 10 days before that runs again and the data gets deleted? Is there a way
to change gc_grace_seconds outside Cassandra.yaml and thus save myself a node
restart?
Note that I am seeing severely degraded performance on this node when it
attempts to compact the HintsColumnFamily to the point where I had to set
setcompactionthroughput to 999 to ensure it doesn't run again (after which the
node started serving requests much faster).
I appreciate the help!
Thanks
