Ah yes sorry be because your behind a raid card. Your need to check the raid config I know on a HP card for example you have an option called enabled disk cache.
This is separate to enabling the raid card cache, the config should be per a drive (is on HP) so worth checking the config outputs for your raid CLI. On Mon, 19 Nov 2018 at 12:47 AM, Alex Litvak <alexander.v.lit...@gmail.com> wrote: > Hmm, > > On all nodes > > hdparm -W /dev/sdb > > /dev/sdb: > SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0d 00 00 00 00 > 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > write-caching = not supported > > > On 11/18/2018 10:30 AM, Ashley Merrick wrote: > > hdparm -W /dev/xxx should show you > > > > On Mon, 19 Nov 2018 at 12:28 AM, Alex Litvak < > alexander.v.lit...@gmail.com <mailto:alexander.v.lit...@gmail.com>> wrote: > > > > All machines state the same. > > > > /opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -DskCache -Lall -a0 > > > > Adapter 0-VD 0(target id: 0): Disk Write Cache : Disk's Default > > Adapter 0-VD 1(target id: 1): Disk Write Cache : Disk's Default > > > > I assume they are all on which is actually bad based on common sense. > > > > > https://notesbytom.wordpress.com/2016/10/21/dell-perc-megaraid-disk-cache-policy/ > > > > An I couldn't find how to confirm it if it is true but vendor > wouldn't ship drives with cache disabled. > > > > I am getting logs in the controller log which are not shown on other > servers > > > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03407a0, localAddr > e03407a0 > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03409e0, localAddr > e03409e0 > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e0340c20, localAddr > e0340c20 > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e0340e60, localAddr > e0340e60 > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03410a0, localAddr > e03410a0 > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03401a0, localAddr > e03401a0 > > > > Not sure if it has any relation to the issue of latency but search > returned nothing substantial. > > > > On 11/18/2018 8:52 AM, Serkan Çoban wrote: > > > I am not saying controller cache, you should check ssd disk > caches. > > > On Sun, Nov 18, 2018 at 11:40 AM Alex Litvak > > > <alexander.v.lit...@gmail.com <mailto: > alexander.v.lit...@gmail.com>> wrote: > > >> > > >> All 3 nodes have this status for SSD mirror. Controller cache > is on for all 3. > > >> > > >> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write > Cache if Bad BBU > > >> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write > Cache if Bad BBU > > >> > > >> > > >> On 11/18/2018 12:45 AM, Serkan Çoban wrote: > > >>> Does write cache on SSDs enabled on three servers? Can you > check them? > > >>> On Sun, Nov 18, 2018 at 9:05 AM Alex Litvak > > >>> <alexander.v.lit...@gmail.com <mailto: > alexander.v.lit...@gmail.com>> wrote: > > >>>> > > >>>> Raid card for journal disks is Perc H730 (Megaraid), RAID 1, > battery back cache is on > > >>>> > > >>>> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No > Write Cache if Bad BBU > > >>>> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No > Write Cache if Bad BBU > > >>>> > > >>>> I have 2 other nodes with older Perc H710 and similar SSDs > with slightly higher wear (6.3% vs 5.18%) but from observation they hardly > hit 1.5 ms on rear occasion > > >>>> Cache, RAID, and battery situation is the same. > > >>>> > > >>>> On 11/17/2018 11:38 PM, Serkan Çoban wrote: > > >>>>>> 10ms w_await for SSD is too much. How that SSD is connected > to the system? Any raid card installed on this system? What is the raid > mode? > > >>>>> On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak > > >>>>> <alexander.v.lit...@gmail.com <mailto: > alexander.v.lit...@gmail.com>> wrote: > > >>>>>> > > >>>>>> Here is another snapshot. I wonder if this write io wait is > too big > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 23.00 0.00 > 336.00 29.22 0.34 14.74 0.00 14.74 2.87 6.60 > > >>>>>> dm-15 0.00 0.00 0.00 16.00 0.00 > 200.00 25.00 0.01 0.75 0.00 0.75 0.75 1.20 > > >>>>>> dm-16 0.00 0.00 0.00 17.00 0.00 > 276.00 32.47 0.25 14.94 0.00 14.94 3.35 5.70 > > >>>>>> dm-17 0.00 0.00 0.00 17.00 0.00 > 252.00 29.65 0.32 18.65 0.00 18.65 4.00 6.80 > > >>>>>> dm-18 0.00 0.00 0.00 15.00 0.00 > 152.00 20.27 0.25 16.80 0.00 16.80 4.07 6.10 > > >>>>>> dm-19 0.00 0.00 0.00 13.00 0.00 > 152.00 23.38 0.21 15.92 0.00 15.92 4.85 6.30 > > >>>>>> dm-20 0.00 0.00 0.00 20.00 0.00 > 248.00 24.80 0.27 13.60 0.00 13.60 3.25 6.50 > > >>>>>> dm-21 0.00 0.00 0.00 17.00 0.00 > 188.00 22.12 0.27 16.00 0.00 16.00 3.59 6.10 > > >>>>>> dm-22 0.00 0.00 0.00 20.00 0.00 > 156.00 15.60 0.11 5.55 0.00 5.55 2.95 5.90 > > >>>>>> dm-24 0.00 0.00 0.00 8.00 0.00 > 56.00 14.00 0.12 14.62 0.00 14.62 4.75 3.80 > > >>>>>> dm-25 0.00 0.00 0.00 19.00 0.00 > 200.00 21.05 0.21 10.89 0.00 10.89 2.74 5.20 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 11.00 0.00 > 136.00 24.73 0.11 9.73 0.00 9.73 1.82 2.00 > > >>>>>> dm-15 0.00 0.00 0.00 12.00 0.00 > 136.00 22.67 0.04 3.75 0.00 3.75 1.08 1.30 > > >>>>>> dm-16 0.00 0.00 0.00 9.00 0.00 > 104.00 23.11 0.09 10.44 0.00 10.44 2.44 2.20 > > >>>>>> dm-17 0.00 0.00 0.00 5.00 0.00 > 160.00 64.00 0.02 4.00 0.00 4.00 4.00 2.00 > > >>>>>> dm-18 0.00 0.00 0.00 5.00 0.00 > 52.00 20.80 0.03 5.80 0.00 5.80 3.60 1.80 > > >>>>>> dm-19 0.00 0.00 0.00 10.00 0.00 > 104.00 20.80 0.08 7.90 0.00 7.90 2.10 2.10 > > >>>>>> dm-20 0.00 0.00 0.00 9.00 0.00 > 132.00 29.33 0.10 11.22 0.00 11.22 2.56 2.30 > > >>>>>> dm-21 0.00 0.00 0.00 6.00 0.00 > 68.00 22.67 0.07 12.33 0.00 12.33 3.83 2.30 > > >>>>>> dm-22 0.00 0.00 0.00 3.00 0.00 > 20.00 13.33 0.01 3.67 0.00 3.67 3.67 1.10 > > >>>>>> dm-24 0.00 0.00 0.00 4.00 0.00 > 24.00 12.00 0.07 18.00 0.00 18.00 5.25 2.10 > > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 > 64.00 21.33 0.06 10.33 0.00 10.33 3.67 2.20 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 5.00 0.00 > 140.00 56.00 0.08 15.20 0.00 15.20 5.40 2.70 > > >>>>>> dm-15 0.00 0.00 0.00 6.00 0.00 > 236.00 78.67 0.18 30.67 0.00 30.67 6.83 4.10 > > >>>>>> dm-16 0.00 0.00 0.00 8.00 0.00 > 84.00 21.00 0.06 7.25 0.00 7.25 1.62 1.30 > > >>>>>> dm-17 0.00 0.00 0.00 3.00 0.00 > 84.00 56.00 0.00 0.33 0.00 0.33 0.33 0.10 > > >>>>>> dm-18 0.00 0.00 0.00 2.00 0.00 > 20.00 20.00 0.02 12.00 0.00 12.00 12.00 2.40 > > >>>>>> dm-19 0.00 0.00 0.00 12.00 0.00 > 80.00 13.33 0.05 4.00 0.00 4.00 2.33 2.80 > > >>>>>> dm-20 0.00 0.00 0.00 16.00 0.00 > 256.00 32.00 0.00 0.06 0.00 0.06 0.06 0.10 > > >>>>>> dm-21 0.00 0.00 0.00 8.00 0.00 > 500.00 125.00 0.00 0.12 0.00 0.12 0.12 0.10 > > >>>>>> dm-22 0.00 0.00 0.00 1.00 0.00 > 8.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-24 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-25 0.00 0.00 0.00 2.00 0.00 > 32.00 32.00 0.08 40.00 0.00 40.00 20.50 4.10 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 10.00 0.00 > 108.00 21.60 0.11 10.80 0.00 10.80 1.90 1.90 > > >>>>>> dm-15 0.00 0.00 0.00 5.00 0.00 > 60.00 24.00 0.03 6.20 0.00 6.20 3.40 1.70 > > >>>>>> dm-16 0.00 0.00 0.00 6.00 0.00 > 68.00 22.67 0.00 0.17 0.00 0.17 0.17 0.10 > > >>>>>> dm-17 0.00 0.00 0.00 7.00 0.00 > 100.00 28.57 0.09 12.29 0.00 12.29 2.86 2.00 > > >>>>>> dm-18 0.00 0.00 0.00 5.00 0.00 > 44.00 17.60 0.07 14.00 0.00 14.00 4.00 2.00 > > >>>>>> dm-19 0.00 0.00 0.00 2.00 0.00 > 24.00 24.00 0.03 14.00 0.00 14.00 8.50 1.70 > > >>>>>> dm-20 0.00 0.00 0.00 7.00 0.00 > 96.00 27.43 0.07 9.57 0.00 9.57 2.00 1.40 > > >>>>>> dm-21 0.00 0.00 0.00 7.00 0.00 > 84.00 24.00 0.12 16.57 0.00 16.57 4.86 3.40 > > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 > 60.00 24.00 0.05 10.20 0.00 10.20 3.40 1.70 > > >>>>>> dm-24 0.00 0.00 0.00 2.00 0.00 > 16.00 16.00 0.03 17.00 0.00 17.00 9.00 1.80 > > >>>>>> dm-25 0.00 0.00 0.00 4.00 0.00 > 76.00 38.00 0.05 11.75 0.00 11.75 4.75 1.90 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 9.00 0.00 > 104.00 23.11 0.14 15.33 0.00 15.33 4.33 3.90 > > >>>>>> dm-15 0.00 0.00 0.00 11.00 0.00 > 128.00 23.27 0.01 0.82 0.00 0.82 0.82 0.90 > > >>>>>> dm-16 0.00 0.00 0.00 10.00 0.00 > 120.00 24.00 0.09 9.30 0.00 9.30 3.30 3.30 > > >>>>>> dm-17 0.00 0.00 0.00 5.00 0.00 > 72.00 28.80 0.04 8.20 0.00 8.20 8.20 4.10 > > >>>>>> dm-18 0.00 0.00 0.00 7.00 0.00 > 68.00 19.43 0.08 11.57 0.00 11.57 6.29 4.40 > > >>>>>> dm-19 0.00 0.00 0.00 7.00 0.00 > 76.00 21.71 0.10 13.86 0.00 13.86 6.29 4.40 > > >>>>>> dm-20 0.00 0.00 0.00 15.00 0.00 > 176.00 23.47 0.12 7.87 0.00 7.87 1.60 2.40 > > >>>>>> dm-21 0.00 0.00 0.00 9.00 0.00 > 112.00 24.89 0.17 18.56 0.00 18.56 6.56 5.90 > > >>>>>> dm-22 0.00 0.00 0.00 4.00 0.00 > 56.00 28.00 0.04 10.75 0.00 10.75 7.50 3.00 > > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 > 532.00 212.80 0.02 4.00 0.00 4.00 3.60 1.80 > > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 > 60.00 20.00 0.11 19.00 0.00 19.00 7.17 4.30 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 9.00 0.00 > 112.00 24.89 0.30 33.22 0.00 33.22 4.33 3.90 > > >>>>>> dm-15 0.00 0.00 0.00 8.00 0.00 > 96.00 24.00 0.08 9.88 0.00 9.88 2.62 2.10 > > >>>>>> dm-16 0.00 0.00 0.00 5.00 0.00 > 92.00 36.80 0.01 1.80 0.00 1.80 1.80 0.90 > > >>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 > 104.00 52.00 0.04 10.00 0.00 10.00 10.00 4.00 > > >>>>>> dm-18 0.00 0.00 0.00 9.00 0.00 > 92.00 20.44 0.11 12.00 0.00 12.00 4.44 4.00 > > >>>>>> dm-19 0.00 0.00 0.00 7.00 0.00 > 64.00 18.29 0.10 14.43 0.00 14.43 4.00 2.80 > > >>>>>> dm-20 0.00 0.00 0.00 7.00 0.00 > 84.00 24.00 0.10 14.00 0.00 14.00 3.71 2.60 > > >>>>>> dm-21 0.00 0.00 0.00 2.00 0.00 > 16.00 16.00 0.08 39.50 0.00 39.50 20.50 4.10 > > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 > 36.00 14.40 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 > 52.00 20.80 0.09 17.20 0.00 17.20 5.40 2.70 > > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 > 48.00 16.00 0.10 16.00 0.00 16.00 5.50 3.30 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 5.00 0.00 > 48.00 19.20 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-15 0.00 0.00 0.00 5.00 0.00 > 44.00 17.60 0.01 3.00 0.00 3.00 3.00 1.50 > > >>>>>> dm-16 0.00 0.00 0.00 3.00 0.00 > 32.00 21.33 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 > 40.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-18 0.00 0.00 0.00 3.00 0.00 > 32.00 21.33 0.04 13.00 0.00 13.00 6.67 2.00 > > >>>>>> dm-19 0.00 0.00 0.00 3.00 0.00 > 28.00 18.67 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-20 0.00 0.00 0.00 6.00 0.00 > 64.00 21.33 0.11 17.67 0.00 17.67 3.17 1.90 > > >>>>>> dm-21 0.00 0.00 0.00 1.00 0.00 > 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-22 0.00 0.00 0.00 1.00 0.00 > 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> dm-24 0.00 0.00 0.00 3.00 0.00 > 20.00 13.33 0.00 0.33 0.00 0.33 0.33 0.10 > > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 > 52.00 17.33 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 18.00 0.00 > 236.00 26.22 0.42 23.28 0.00 23.28 3.72 6.70 > > >>>>>> dm-15 0.00 0.00 0.00 15.00 0.00 > 240.00 32.00 0.12 8.27 0.00 8.27 2.00 3.00 > > >>>>>> dm-16 0.00 0.00 0.00 10.00 0.00 > 120.00 24.00 0.23 22.80 0.00 22.80 6.50 6.50 > > >>>>>> dm-17 0.00 0.00 0.00 6.00 0.00 > 68.00 22.67 0.06 9.83 0.00 9.83 5.00 3.00 > > >>>>>> dm-18 0.00 0.00 0.00 12.00 0.00 > 152.00 25.33 0.10 8.17 0.00 8.17 3.17 3.80 > > >>>>>> dm-19 0.00 0.00 0.00 12.00 0.00 > 136.00 22.67 0.14 11.58 0.00 11.58 3.42 4.10 > > >>>>>> dm-20 0.00 0.00 0.00 19.00 0.00 > 296.00 31.16 0.20 10.63 0.00 10.63 1.84 3.50 > > >>>>>> dm-21 0.00 0.00 0.00 14.00 0.00 > 132.00 18.86 0.17 12.14 0.00 12.14 3.71 5.20 > > >>>>>> dm-22 0.00 0.00 0.00 10.00 0.00 > 84.00 16.80 0.12 12.00 0.00 12.00 6.60 6.60 > > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 > 40.00 16.00 0.10 19.40 0.00 19.40 9.80 4.90 > > >>>>>> dm-25 0.00 0.00 0.00 9.00 0.00 > 152.00 33.78 0.08 9.33 0.00 9.33 3.44 3.10 > > >>>>>> > > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>> dm-14 0.00 0.00 0.00 8.00 0.00 > 112.00 28.00 0.08 9.75 0.00 9.75 3.88 3.10 > > >>>>>> dm-15 0.00 0.00 0.00 14.00 0.00 > 156.00 22.29 0.13 9.43 0.00 9.43 2.71 3.80 > > >>>>>> dm-16 0.00 0.00 0.00 12.00 0.00 > 132.00 22.00 0.11 8.83 0.00 8.83 2.17 2.60 > > >>>>>> dm-17 0.00 0.00 0.00 8.00 0.00 > 116.00 29.00 0.08 10.12 0.00 10.12 4.25 3.40 > > >>>>>> dm-18 0.00 0.00 0.00 8.00 0.00 > 100.00 25.00 0.08 9.62 0.00 9.62 5.00 4.00 > > >>>>>> dm-19 0.00 0.00 0.00 20.00 0.00 > 212.00 21.20 0.25 12.75 0.00 12.75 2.60 5.20 > > >>>>>> dm-20 0.00 0.00 0.00 16.00 0.00 > 168.00 21.00 0.04 2.69 0.00 2.69 0.81 1.30 > > >>>>>> dm-21 0.00 0.00 0.00 5.00 0.00 > 48.00 19.20 0.05 10.80 0.00 10.80 7.80 3.90 > > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 > 52.00 20.80 0.03 6.80 0.00 6.80 3.40 1.70 > > >>>>>> dm-24 0.00 0.00 0.00 8.00 0.00 > 60.00 15.00 0.09 11.75 0.00 11.75 2.25 1.80 > > >>>>>> dm-25 0.00 0.00 0.00 12.00 0.00 > 160.00 26.67 0.06 5.08 0.00 5.08 1.25 1.50 > > >>>>>> > > >>>>>> On 11/17/2018 10:19 PM, Alex Litvak wrote: > > >>>>>>> I stand corrected, I looked at the device iostat, but it > was partitioned. Here is a more correct picture of what is going on now. > > >>>>>>> > > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>>> dm-14 0.00 0.00 0.00 19.00 0.00 > 4116.00 433.26 0.01 0.68 0.00 0.68 0.05 0.10 > > >>>>>>> dm-15 0.00 0.00 0.00 35.00 0.00 > 8224.00 469.94 0.03 0.86 0.00 0.86 0.06 0.20 > > >>>>>>> dm-16 0.00 0.00 0.00 53.00 0.00 > 12428.00 468.98 0.11 2.04 0.00 2.04 0.17 0.90 > > >>>>>>> dm-17 0.00 0.00 0.00 43.00 0.00 > 8344.00 388.09 0.09 2.14 0.00 2.14 0.42 1.80 > > >>>>>>> dm-18 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-19 0.00 0.00 0.00 75.00 0.00 > 16824.00 448.64 0.08 1.11 0.00 1.11 0.08 0.60 > > >>>>>>> dm-20 0.00 0.00 0.00 70.00 0.00 > 16452.00 470.06 0.06 0.90 0.00 0.90 0.09 0.60 > > >>>>>>> dm-21 0.00 0.00 0.00 18.00 0.00 > 4112.00 456.89 0.02 1.00 0.00 1.00 0.11 0.20 > > >>>>>>> dm-22 0.00 0.00 0.00 53.00 0.00 > 12324.00 465.06 0.06 0.70 0.00 0.70 0.08 0.40 > > >>>>>>> dm-24 0.00 0.00 0.00 18.00 0.00 > 4272.00 474.67 0.02 1.06 0.00 1.06 0.17 0.30 > > >>>>>>> dm-25 0.00 0.00 0.00 74.00 0.00 > 16916.00 457.19 0.09 1.26 0.00 1.26 0.18 1.30 > > >>>>>>> > > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>>> dm-14 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-15 0.00 0.00 0.00 17.00 0.00 > 4108.00 483.29 0.02 1.00 0.00 1.00 0.06 0.10 > > >>>>>>> dm-16 0.00 0.00 0.00 34.00 0.00 > 8208.00 482.82 0.03 1.00 0.00 1.00 0.06 0.20 > > >>>>>>> dm-17 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-18 0.00 0.00 0.00 36.00 0.00 > 8220.00 456.67 0.05 1.33 0.00 1.33 0.08 0.30 > > >>>>>>> dm-19 0.00 0.00 0.00 1.00 0.00 > 8.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-20 0.00 0.00 0.00 36.00 0.00 > 8288.00 460.44 0.05 1.42 0.00 1.42 0.08 0.30 > > >>>>>>> dm-21 0.00 0.00 0.00 34.00 0.00 > 8208.00 482.82 0.03 1.00 0.00 1.00 0.06 0.20 > > >>>>>>> dm-22 0.00 0.00 0.00 18.00 0.00 > 4128.00 458.67 0.04 3.22 0.00 3.22 0.17 0.30 > > >>>>>>> dm-24 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> > > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>>> dm-14 0.00 0.00 0.00 20.00 0.00 > 4032.00 403.20 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-15 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-16 0.00 0.00 0.00 1.00 0.00 > 20.00 40.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 > 28.00 14.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-18 0.00 0.00 0.00 3.00 0.00 > 36.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-19 0.00 0.00 0.00 2.00 0.00 > 20.00 20.00 0.01 2.50 0.00 2.50 2.50 0.50 > > >>>>>>> dm-20 0.00 0.00 0.00 6.00 0.00 > 96.00 32.00 0.02 3.33 0.00 3.33 2.00 1.20 > > >>>>>>> dm-21 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-22 0.00 0.00 0.00 2.00 0.00 > 32.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-24 0.00 0.00 0.00 22.00 0.00 > 4184.00 380.36 0.10 4.59 0.00 4.59 0.95 2.10 > > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> > > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s > wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > > >>>>>>> dm-14 0.00 0.00 0.00 8.00 0.00 > 1928.00 482.00 0.01 1.00 0.00 1.00 0.12 0.10 > > >>>>>>> dm-15 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-16 0.00 0.00 0.00 3.00 0.00 > 312.00 208.00 0.00 0.33 0.00 0.33 0.33 0.10 > > >>>>>>> dm-17 0.00 0.00 0.00 18.00 0.00 > 4264.00 473.78 0.03 1.67 0.00 1.67 0.11 0.20 > > >>>>>>> dm-18 0.00 0.00 0.00 17.00 0.00 > 4104.00 482.82 0.03 1.82 0.00 1.82 0.12 0.20 > > >>>>>>> dm-19 0.00 0.00 0.00 18.00 0.00 > 4112.00 456.89 0.02 1.06 0.00 1.06 0.11 0.20 > > >>>>>>> dm-20 0.00 0.00 0.00 32.00 0.00 > 4308.00 269.25 0.03 0.81 0.00 0.81 0.34 1.10 > > >>>>>>> dm-21 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-22 0.00 0.00 0.00 8.00 0.00 > 540.00 135.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> dm-24 0.00 0.00 0.00 35.00 0.00 > 8228.00 470.17 0.03 0.97 0.00 0.97 0.06 0.20 > > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On 11/17/2018 5:27 PM, John Petrini wrote: > > >>>>>>>> The iostat isn't very helpful because there are not many > writes. I'd recommend disabling cstates entirely, not sure it's your > problem but it's good practice and if your cluster goes as > > idle as your > > >>>>>>>> iostat suggests it could be the culprit. > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
