heap is the normal proportional, probably 1/2 RAM. So there definitely will be larger non-heap for file caching.
The Amy Tobey utility does not show churn in the file cache however, which if it was almost three orders of magnitude difference in amount of disk access, I would expect churn in the OS file cache. I cannot find the files in the data dirs that are being accessed that are different. If there is massive differences in file read I/O, I would ASSUME there will be file reads detectable with inotifywait, but nothing was seen. So if anyone has black magic CLI incantations on linux/ubuntu to help track the files being actively accessed/read by a process (lsof is hard to follow with the thread vs pid obfuscation) On Wed, Dec 2, 2020 at 3:10 PM Elliott Sims <elli...@backblaze.com> wrote: > Is the heap larger on the M5.4x instance? > Are you sure it's Cassandra generating the read traffic vs just evicting > files read by other systems? > > In general, I'd call "more RAM means fewer drive reads" a very expected > result regardless of the details, especially when it's the difference > between fitting the whole data-set in RAM and not, so I'm not sure it's > worth doing that much digging. > > > On Wed, Dec 2, 2020 at 8:41 AM Carl Mueller > <carl.muel...@smartthings.com.invalid> wrote: > >> Oh, this is cassandra 2.2.13 (multi tenant delays) and ubuntu 18.04. >> >> On Wed, Dec 2, 2020 at 10:35 AM Carl Mueller < >> carl.muel...@smartthings.com> wrote: >> >>> We have a cluster that is experiencing very high disk read I/O in the >>> 20-40 MB/sec range on m5.2x (gp2 drives). This is verified via VM metrics >>> as well as iotop. >>> >>> When we switch m5.4x it drops to 60 KB/sec. >>> >>> There is no difference in network send/recv, read/write request counts. >>> >>> The graph for read kb/sec mirrors the cpu.iowait. >>> >>> Compaction would have similar writes to go with reads as the sstables >>> were written. Flushing would be almost all writes. Swappiness is zero. >>> >>> I have done inotifywait to compare read volume on the data and log dirs. >>> They are roughly equivalent. >>> >>> File Caching could be a candidate, I used tobert's : >>> https://github.com/tobert/pcstat to see what files are in the file >>> cache, and that listed all files at 100%, I would think an overloaded file >>> cache would have different files swapping into the cache and partials on >>> the data files (data density for the node is about 30 GB). >>> >>> iotop indicates all the read traffic is from cassandra threads. >>> >>> Anyone have similar experiences? >>> >>
