Hi Experts, Seeking your help on a production issue. We were running high write intensive job on our 3 node cassandra cluster V 2.1.7.
TPS on nodes were high. Job ran for more than 2 days and thereafter, loadavg on 1 of the node increased to very high number like loadavg : 29. System log reports: INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 2 READ messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last 5000ms The job was stopped due to heavy load. But sill after 12 hours , we can see mutation drops messages and sudden increase on avgload Are these hintedhandoff mutations? Can we stop these. Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any load or any such activity. Due to heavy load and GC , there are intermittent gossip failures among node. Can you someone Please help. PS: Load job was stopped on cluster. Everything ran fine for few hours and and Later issue started again like mutation messages drops. Thanks and Regards, Varun Saluja --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
