Hi All, I have a 44 node cluster (22 nodes on each DC). Each node has 24 cores and 130 GB RAM, 3 TB HDDs. Version 2.0.14 (soon to be upgraded) ~10K writes per second per node. Heap size: 8 GB max, 2.4 GB newgen
I deployed Reaper and GC started to increase rapidly. I'm not sure if it's because there was a lot of inconsistency in the data, but I decided to increase the heap to 16 GB and new gen to 6 GB. I increased the max tenure from 1 to 5. I tested on a canary node and everything was fine but when I changed the entire DC, I suddenly saw a lot of dropped mutations in the logs on most of the nodes. (Reaper was not running on the cluster yet but a manual repair was running). Can the heap increment cause lots of dropped mutations? When is a mutation considered as dropped? Is it during flush? Is it during the write to the commit log or memtable? Thanks!
