How is client makes call to Cassandra, is they load balancer, Can we look at load balancer log. This usually captures and respond for no of client calls per second. Ask them to share their monitoring chart
If your question is why Cassandra load went up, there could be multiple reasons, balancer command, mutations, or node crashed and came up, or simply network blip issue Thank you Sent from my iPhone > On Oct 25, 2021, at 5:18 AM, Bowen Song <bo...@bso.ng> wrote: > > > For older version (Cassandra < 4.0) you can use `nodetool > settraceprobability` to get a sample of queries (or all queries, if your > cluster can handle the extra load). > > For newer version (>= 4.0) you can use the above, or the new Full Query > Logging. > > Alternatively, tcpdump (if no SSL) and strace can be used to analyse a > specific client server and/or process. > > > >> On 25/10/2021 04:54, S G wrote: >> Hello, >> >> We recently faced an issue recently where the read traffic on a big >> Cassandra cluster shot up several times (think more than 20 times). >> >> However, the client team denies sending any huge load and they have their >> own traffic graphs to prove the same. >> >> Assuming the client team's graphs are correct, how do we know the source of >> traffic ? Slow query logging is enabled, but it only logs queries after a >> certain threshold, so not very helpful. >> >> Secondly, we do not know when the incidence will re-occur. So how do we >> solve such a problem and put some monitoring in place that shows the source >> of such huge spikes when it happens next time. >> >> Thinking of trying lsof -i and netstat -tn commands in a per-minute cron on >> each server but they only show connections from clients, not how many >> requests in those connections. >> Any suggestions on how to go about this? >> >> Thanks ! >>
