The IOPS dropped with the drop in read I/O throughput. The cassandra reads and network sent/revd is the same.
We also did not adjust our heap size at 2x. between cachestats and thinking about how mmap works (probably doesn't formally access files in a way inotify monitors would detect) and given our node size would fit into the 4x's excess ram, I'm pretty sure it's a starved file cache. It just seems like we went for zero file cache hits to 100%. On Wed, Dec 2, 2020 at 6:16 PM Jeff Jirsa <jji...@gmail.com> wrote: > What isn't clear is whether or not the slow IO is a result of doing far > fewer IO and serving from RAM or if slow IO is slow IO on same number of > reads/second. I assume you're doing far fewer IO, and the slowness is a > sampling error. > > Do you know how many ops/second you're reading from each disk? Cloudwatch > would have that. > > If you dont want to actually inspect the system, run a program that holds > half the memory, to put yourself down at the same memory level as m5.2xl, > see if the lack of page cache causes disk IO to return. > > > > On Wed, Dec 2, 2020 at 2:56 PM Carl Mueller > <carl.muel...@smartthings.com.invalid> wrote: > >> I agree in theory, I just want some way to confirm that file accesses in >> the larger instance are being intercepted by the file cache, vs what is >> happening in the other case. >> >> I've tried amy tobey's pcstat >> >> I'd assume the 2x would have a file cache with lots of partial caches of >> files, churn in the files that are cached. But all the Index.db and Data.db >> files are 100% cached according to that tool >> >> On Wed, Dec 2, 2020 at 4:53 PM Jeff Jirsa <jji...@gmail.com> wrote: >> >>> This is exactly what I would expect when you double the memory and all >>> of the data lives in page cache. >>> >>> >>> On Wed, Dec 2, 2020 at 8:41 AM Carl Mueller >>> <carl.muel...@smartthings.com.invalid> wrote: >>> >>>> Oh, this is cassandra 2.2.13 (multi tenant delays) and ubuntu 18.04. >>>> >>>> On Wed, Dec 2, 2020 at 10:35 AM Carl Mueller < >>>> carl.muel...@smartthings.com> wrote: >>>> >>>>> We have a cluster that is experiencing very high disk read I/O in the >>>>> 20-40 MB/sec range on m5.2x (gp2 drives). This is verified via VM metrics >>>>> as well as iotop. >>>>> >>>>> When we switch m5.4x it drops to 60 KB/sec. >>>>> >>>>> There is no difference in network send/recv, read/write request >>>>> counts. >>>>> >>>>> The graph for read kb/sec mirrors the cpu.iowait. >>>>> >>>>> Compaction would have similar writes to go with reads as the sstables >>>>> were written. Flushing would be almost all writes. Swappiness is zero. >>>>> >>>>> I have done inotifywait to compare read volume on the data and log >>>>> dirs. They are roughly equivalent. >>>>> >>>>> File Caching could be a candidate, I used tobert's : >>>>> https://github.com/tobert/pcstat to see what files are in the file >>>>> cache, and that listed all files at 100%, I would think an overloaded file >>>>> cache would have different files swapping into the cache and partials on >>>>> the data files (data density for the node is about 30 GB). >>>>> >>>>> iotop indicates all the read traffic is from cassandra threads. >>>>> >>>>> Anyone have similar experiences? >>>>> >>>>
