What isn't clear is whether or not the slow IO is a result of doing far fewer IO and serving from RAM or if slow IO is slow IO on same number of reads/second. I assume you're doing far fewer IO, and the slowness is a sampling error.
Do you know how many ops/second you're reading from each disk? Cloudwatch would have that. If you dont want to actually inspect the system, run a program that holds half the memory, to put yourself down at the same memory level as m5.2xl, see if the lack of page cache causes disk IO to return. On Wed, Dec 2, 2020 at 2:56 PM Carl Mueller <carl.muel...@smartthings.com.invalid> wrote: > I agree in theory, I just want some way to confirm that file accesses in > the larger instance are being intercepted by the file cache, vs what is > happening in the other case. > > I've tried amy tobey's pcstat > > I'd assume the 2x would have a file cache with lots of partial caches of > files, churn in the files that are cached. But all the Index.db and Data.db > files are 100% cached according to that tool > > On Wed, Dec 2, 2020 at 4:53 PM Jeff Jirsa <jji...@gmail.com> wrote: > >> This is exactly what I would expect when you double the memory and all of >> the data lives in page cache. >> >> >> On Wed, Dec 2, 2020 at 8:41 AM Carl Mueller >> <carl.muel...@smartthings.com.invalid> wrote: >> >>> Oh, this is cassandra 2.2.13 (multi tenant delays) and ubuntu 18.04. >>> >>> On Wed, Dec 2, 2020 at 10:35 AM Carl Mueller < >>> carl.muel...@smartthings.com> wrote: >>> >>>> We have a cluster that is experiencing very high disk read I/O in the >>>> 20-40 MB/sec range on m5.2x (gp2 drives). This is verified via VM metrics >>>> as well as iotop. >>>> >>>> When we switch m5.4x it drops to 60 KB/sec. >>>> >>>> There is no difference in network send/recv, read/write request counts. >>>> >>>> The graph for read kb/sec mirrors the cpu.iowait. >>>> >>>> Compaction would have similar writes to go with reads as the sstables >>>> were written. Flushing would be almost all writes. Swappiness is zero. >>>> >>>> I have done inotifywait to compare read volume on the data and log >>>> dirs. They are roughly equivalent. >>>> >>>> File Caching could be a candidate, I used tobert's : >>>> https://github.com/tobert/pcstat to see what files are in the file >>>> cache, and that listed all files at 100%, I would think an overloaded file >>>> cache would have different files swapping into the cache and partials on >>>> the data files (data density for the node is about 30 GB). >>>> >>>> iotop indicates all the read traffic is from cassandra threads. >>>> >>>> Anyone have similar experiences? >>>> >>>
