I agree in theory, I just want some way to confirm that file accesses in the larger instance are being intercepted by the file cache, vs what is happening in the other case.
I've tried amy tobey's pcstat I'd assume the 2x would have a file cache with lots of partial caches of files, churn in the files that are cached. But all the Index.db and Data.db files are 100% cached according to that tool On Wed, Dec 2, 2020 at 4:53 PM Jeff Jirsa <jji...@gmail.com> wrote: > This is exactly what I would expect when you double the memory and all of > the data lives in page cache. > > > On Wed, Dec 2, 2020 at 8:41 AM Carl Mueller > <carl.muel...@smartthings.com.invalid> wrote: > >> Oh, this is cassandra 2.2.13 (multi tenant delays) and ubuntu 18.04. >> >> On Wed, Dec 2, 2020 at 10:35 AM Carl Mueller < >> carl.muel...@smartthings.com> wrote: >> >>> We have a cluster that is experiencing very high disk read I/O in the >>> 20-40 MB/sec range on m5.2x (gp2 drives). This is verified via VM metrics >>> as well as iotop. >>> >>> When we switch m5.4x it drops to 60 KB/sec. >>> >>> There is no difference in network send/recv, read/write request counts. >>> >>> The graph for read kb/sec mirrors the cpu.iowait. >>> >>> Compaction would have similar writes to go with reads as the sstables >>> were written. Flushing would be almost all writes. Swappiness is zero. >>> >>> I have done inotifywait to compare read volume on the data and log dirs. >>> They are roughly equivalent. >>> >>> File Caching could be a candidate, I used tobert's : >>> https://github.com/tobert/pcstat to see what files are in the file >>> cache, and that listed all files at 100%, I would think an overloaded file >>> cache would have different files swapping into the cache and partials on >>> the data files (data density for the node is about 30 GB). >>> >>> iotop indicates all the read traffic is from cassandra threads. >>> >>> Anyone have similar experiences? >>> >>
