Hello, I have my 3-node, RF=3 cluster acting strangely. Can someone shed a light as to what is going on ? It was stuck for a couple of hours (all clients TimedOut). nodetool tpstats showed huge increasing MutationStages (in the hundreds of thousands). I restarted one node and it took a while to reply GB of commitlog. I've shutdown all clients that write to the cluster and it's just weird
All nodes are still showing huge MutationStages including the new one and it's either increasing or stable. The pending count is stuck at 32. Compactionstats shows no compaction on 2 nodes and dozens of Scrub compactions (all at 100%) on the 3rd one. This is a scrub I did last week when I encountered assertion errors. Netstats shows no streams being exchanged at any node but each on is expecting a few Responses. Any ideas ? Thanks For example (increased to 567062 while I was writing this email) Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 18372664517 0 0 RequestResponseStage 0 0 10731370183 0 0 MutationStage 32 565879 295492216 0 0 ReadRepairStage 0 0 23654 0 0 ReplicateOnWriteStage 0 0 7733659 0 0 GossipStage 0 0 3502922 0 0 AntiEntropyStage 0 0 1631 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlusher 0 0 5716 0 0 StreamStage 0 0 10 0 0 FlushWriter 0 0 5714 0 499 FILEUTILS-DELETE-POOL 0 0 773 0 0 MiscStage 0 0 1266 0 0 FlushSorter 0 0 0 0 0 AntiEntropySessions 0 0 18 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 1798 0 0 Mode: Normal Not sending any streams. Not receiving any streams. Pool Name Active Pending Completed Commands n/a 0 1223769753 Responses n/a 4 1627481305
