If you've got small partitions/small reads you should test lowering your compression chunk size on the table and disabling read ahead. This sounds like it might just be a case of read amplification.
On Tue., 8 May 2018, 05:43 Kyrylo Lebediev, <kyrylo_lebed...@epam.com> wrote: > Dear Experts, > > > I'm observing strange behavior on a cluster 2.1.20 during compactions. > > > My setup is: > > 12 nodes m4.2xlarge (8 vCPU, 32G RAM) Ubuntu 16.04, 2T EBS gp2. > > Filesystem: XFS, blocksize 4k, device read-ahead - 4k > > /sys/block/vxdb/queue/nomerges = 0 > > SizeTieredCompactionStrategy > > > After data loads when effectively nothing else is talking to the cluster > and compactions is the only activity, I see something like this: > $ iostat -dkx 1 > ... > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > xvda 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > xvdb 0.00 0.00 4769.00 213.00 19076.00 26820.00 > 18.42 7.95 1.17 1.06 3.76 0.20 100.00 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > xvda 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > xvdb 0.00 0.00 6098.00 177.00 24392.00 22076.00 > 14.81 6.46 1.36 0.96 15.16 0.16 100.00 > > Writes are fine: 177 writes/sec <-> ~22Mbytes/sec, > > But for some reason compactions generate a huge number of small reads: > 6098 reads/s <-> ~24Mbytes/sec. ===> Read size is 4k > > > Why instead much smaller amount of large reads I'm getting huge number of > 4k reads instead? > > What could be the reason? > > Thanks, > > Kyrill > > >
