What is your replication factor and write consistency? :) On Sep 23, 2015 7:28 AM, "Björn Hachmann" <bjoern.hachm...@metrigo.de> wrote:
> Today I realized that one of the nodes in our Cassandra cluster (2.1.7) is > storing a lot of hints (>80GB) and I fail to see a convincing way to deal > with them. > > From the system.log: > INFO [ScheduledTasks:1] 2015-09-23 14:27:06,692 StatusLogger.java:115 - > system.hints 276,1010945 > INFO [ScheduledTasks:1] 2015-09-23 14:38:06,722 StatusLogger.java:115 - > system.hints 968,2968163 > INFO [ScheduledTasks:1] 2015-09-23 14:38:41,742 StatusLogger.java:115 - > system.hints 1317,3799471 > INFO [ScheduledTasks:1] 2015-09-23 14:49:16,775 StatusLogger.java:115 - > system.hints 1519,4399905 > INFO [ScheduledTasks:1] 2015-09-23 14:49:36,793 StatusLogger.java:115 - > system.hints 2247,6514649 > INFO [ScheduledTasks:1] 2015-09-23 14:49:41,811 StatusLogger.java:115 - > system.hints 2247,6514649 > INFO [ScheduledTasks:1] 2015-09-23 14:49:51,830 StatusLogger.java:115 - > system.hints 2368,6733293 > INFO [ScheduledTasks:1] 2015-09-23 15:00:41,885 StatusLogger.java:115 - > system.hints 283,450166810 > INFO [ScheduledTasks:1] 2015-09-23 15:12:16,919 StatusLogger.java:115 - > system.hints 232,970964 > INFO [ScheduledTasks:1] 2015-09-23 15:12:31,934 StatusLogger.java:115 - > system.hints 581,2034388 > INFO [ScheduledTasks:1] 2015-09-23 15:23:46,973 StatusLogger.java:115 - > system.hints 234,321566 > INFO [ScheduledTasks:1] 2015-09-23 15:24:01,988 StatusLogger.java:115 - > system.hints 368,935634 > INFO [ScheduledTasks:1] 2015-09-23 15:35:12,039 StatusLogger.java:115 - > system.hints 264,636164 > > The state of the cluster seems stable, at least we do not have any > downtimes (sometimes the load on one of the nodes is quite high). > > We had a look into the table system.hints and from there we learnt that > most hints > are for one of the nodes in our 2nd datacenter and most of the mutations > are > increments to one of our counter tables which are very frequent. > > There seem to be no other suspicious log messages in the log apart from a > few dropped events. > > We have several questions: > - What could be the reason that only one of the nodes has hints for only > one target node, altough every other node should be coordinator for these > queries sometimes also? > - Is there a way to turn of hinted handoff on a table level or on data > center level? > - What could we do to investigate the cause of this issue deeper? > > Thank you! > Kind regards > Björn Hachmann >
