Hello Cassandra community, I see the following log lines in the Cassandra nodes of 2 datacenters that participate in a cluster of 6 datacenters.
INFO [ScheduledTasks:1] 2022-09-12 17:19:44,422 MessagingService.java:1236 - MUTATION messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 9140 ms >From the information I have found online, I see that those logs indicate a >resource issue on the Cassandra nodes. I am trying to troubleshoot what could >be the bottleneck. Can you, please, help me interpret the log line. So far I know that MUTATION means that it is write operations for already existing keys. What internal means? Does it indicate that the write operation is originated from a coordinator node residing in the same datacenter? What cross node means? Does it indicate that the write operation is originated from a coordinator node residing in another datacenter? What dropped latency means? Is it the time that the message spent in some queue before dropped? I would like to know that so as to understand the impact observed on client application. Thank you in advance!!! I find really useful all the information found in different posts!!! BR Michail
