All 3 nodes have this status for SSD mirror. Controller cache is on for all 3.
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
On 11/18/2018 12:45 AM, Serkan Çoban wrote:
Does write cache on SSDs enabled on three servers? Can you check them?
On Sun, Nov 18, 2018 at 9:05 AM Alex Litvak
<alexander.v.lit...@gmail.com> wrote:
Raid card for journal disks is Perc H730 (Megaraid), RAID 1, battery back cache
is on
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
I have 2 other nodes with older Perc H710 and similar SSDs with slightly
higher wear (6.3% vs 5.18%) but from observation they hardly hit 1.5 ms on rear
occasion
Cache, RAID, and battery situation is the same.
On 11/17/2018 11:38 PM, Serkan Çoban wrote:
10ms w_await for SSD is too much. How that SSD is connected to the system? Any
raid card installed on this system? What is the raid mode?
On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak
<alexander.v.lit...@gmail.com> wrote:
Here is another snapshot. I wonder if this write io wait is too big
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 23.00 0.00 336.00 29.22
0.34 14.74 0.00 14.74 2.87 6.60
dm-15 0.00 0.00 0.00 16.00 0.00 200.00 25.00
0.01 0.75 0.00 0.75 0.75 1.20
dm-16 0.00 0.00 0.00 17.00 0.00 276.00 32.47
0.25 14.94 0.00 14.94 3.35 5.70
dm-17 0.00 0.00 0.00 17.00 0.00 252.00 29.65
0.32 18.65 0.00 18.65 4.00 6.80
dm-18 0.00 0.00 0.00 15.00 0.00 152.00 20.27
0.25 16.80 0.00 16.80 4.07 6.10
dm-19 0.00 0.00 0.00 13.00 0.00 152.00 23.38
0.21 15.92 0.00 15.92 4.85 6.30
dm-20 0.00 0.00 0.00 20.00 0.00 248.00 24.80
0.27 13.60 0.00 13.60 3.25 6.50
dm-21 0.00 0.00 0.00 17.00 0.00 188.00 22.12
0.27 16.00 0.00 16.00 3.59 6.10
dm-22 0.00 0.00 0.00 20.00 0.00 156.00 15.60
0.11 5.55 0.00 5.55 2.95 5.90
dm-24 0.00 0.00 0.00 8.00 0.00 56.00 14.00
0.12 14.62 0.00 14.62 4.75 3.80
dm-25 0.00 0.00 0.00 19.00 0.00 200.00 21.05
0.21 10.89 0.00 10.89 2.74 5.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 11.00 0.00 136.00 24.73
0.11 9.73 0.00 9.73 1.82 2.00
dm-15 0.00 0.00 0.00 12.00 0.00 136.00 22.67
0.04 3.75 0.00 3.75 1.08 1.30
dm-16 0.00 0.00 0.00 9.00 0.00 104.00 23.11
0.09 10.44 0.00 10.44 2.44 2.20
dm-17 0.00 0.00 0.00 5.00 0.00 160.00 64.00
0.02 4.00 0.00 4.00 4.00 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.03 5.80 0.00 5.80 3.60 1.80
dm-19 0.00 0.00 0.00 10.00 0.00 104.00 20.80
0.08 7.90 0.00 7.90 2.10 2.10
dm-20 0.00 0.00 0.00 9.00 0.00 132.00 29.33
0.10 11.22 0.00 11.22 2.56 2.30
dm-21 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.07 12.33 0.00 12.33 3.83 2.30
dm-22 0.00 0.00 0.00 3.00 0.00 20.00 13.33
0.01 3.67 0.00 3.67 3.67 1.10
dm-24 0.00 0.00 0.00 4.00 0.00 24.00 12.00
0.07 18.00 0.00 18.00 5.25 2.10
dm-25 0.00 0.00 0.00 6.00 0.00 64.00 21.33
0.06 10.33 0.00 10.33 3.67 2.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 140.00 56.00
0.08 15.20 0.00 15.20 5.40 2.70
dm-15 0.00 0.00 0.00 6.00 0.00 236.00 78.67
0.18 30.67 0.00 30.67 6.83 4.10
dm-16 0.00 0.00 0.00 8.00 0.00 84.00 21.00
0.06 7.25 0.00 7.25 1.62 1.30
dm-17 0.00 0.00 0.00 3.00 0.00 84.00 56.00
0.00 0.33 0.00 0.33 0.33 0.10
dm-18 0.00 0.00 0.00 2.00 0.00 20.00 20.00
0.02 12.00 0.00 12.00 12.00 2.40
dm-19 0.00 0.00 0.00 12.00 0.00 80.00 13.33
0.05 4.00 0.00 4.00 2.33 2.80
dm-20 0.00 0.00 0.00 16.00 0.00 256.00 32.00
0.00 0.06 0.00 0.06 0.06 0.10
dm-21 0.00 0.00 0.00 8.00 0.00 500.00 125.00
0.00 0.12 0.00 0.12 0.12 0.10
dm-22 0.00 0.00 0.00 1.00 0.00 8.00 16.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 2.00 0.00 32.00 32.00
0.08 40.00 0.00 40.00 20.50 4.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 10.00 0.00 108.00 21.60
0.11 10.80 0.00 10.80 1.90 1.90
dm-15 0.00 0.00 0.00 5.00 0.00 60.00 24.00
0.03 6.20 0.00 6.20 3.40 1.70
dm-16 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.00 0.17 0.00 0.17 0.17 0.10
dm-17 0.00 0.00 0.00 7.00 0.00 100.00 28.57
0.09 12.29 0.00 12.29 2.86 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 44.00 17.60
0.07 14.00 0.00 14.00 4.00 2.00
dm-19 0.00 0.00 0.00 2.00 0.00 24.00 24.00
0.03 14.00 0.00 14.00 8.50 1.70
dm-20 0.00 0.00 0.00 7.00 0.00 96.00 27.43
0.07 9.57 0.00 9.57 2.00 1.40
dm-21 0.00 0.00 0.00 7.00 0.00 84.00 24.00
0.12 16.57 0.00 16.57 4.86 3.40
dm-22 0.00 0.00 0.00 5.00 0.00 60.00 24.00
0.05 10.20 0.00 10.20 3.40 1.70
dm-24 0.00 0.00 0.00 2.00 0.00 16.00 16.00
0.03 17.00 0.00 17.00 9.00 1.80
dm-25 0.00 0.00 0.00 4.00 0.00 76.00 38.00
0.05 11.75 0.00 11.75 4.75 1.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 104.00 23.11
0.14 15.33 0.00 15.33 4.33 3.90
dm-15 0.00 0.00 0.00 11.00 0.00 128.00 23.27
0.01 0.82 0.00 0.82 0.82 0.90
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00
0.09 9.30 0.00 9.30 3.30 3.30
dm-17 0.00 0.00 0.00 5.00 0.00 72.00 28.80
0.04 8.20 0.00 8.20 8.20 4.10
dm-18 0.00 0.00 0.00 7.00 0.00 68.00 19.43
0.08 11.57 0.00 11.57 6.29 4.40
dm-19 0.00 0.00 0.00 7.00 0.00 76.00 21.71
0.10 13.86 0.00 13.86 6.29 4.40
dm-20 0.00 0.00 0.00 15.00 0.00 176.00 23.47
0.12 7.87 0.00 7.87 1.60 2.40
dm-21 0.00 0.00 0.00 9.00 0.00 112.00 24.89
0.17 18.56 0.00 18.56 6.56 5.90
dm-22 0.00 0.00 0.00 4.00 0.00 56.00 28.00
0.04 10.75 0.00 10.75 7.50 3.00
dm-24 0.00 0.00 0.00 5.00 0.00 532.00 212.80
0.02 4.00 0.00 4.00 3.60 1.80
dm-25 0.00 0.00 0.00 6.00 0.00 60.00 20.00
0.11 19.00 0.00 19.00 7.17 4.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 112.00 24.89
0.30 33.22 0.00 33.22 4.33 3.90
dm-15 0.00 0.00 0.00 8.00 0.00 96.00 24.00
0.08 9.88 0.00 9.88 2.62 2.10
dm-16 0.00 0.00 0.00 5.00 0.00 92.00 36.80
0.01 1.80 0.00 1.80 1.80 0.90
dm-17 0.00 0.00 0.00 4.00 0.00 104.00 52.00
0.04 10.00 0.00 10.00 10.00 4.00
dm-18 0.00 0.00 0.00 9.00 0.00 92.00 20.44
0.11 12.00 0.00 12.00 4.44 4.00
dm-19 0.00 0.00 0.00 7.00 0.00 64.00 18.29
0.10 14.43 0.00 14.43 4.00 2.80
dm-20 0.00 0.00 0.00 7.00 0.00 84.00 24.00
0.10 14.00 0.00 14.00 3.71 2.60
dm-21 0.00 0.00 0.00 2.00 0.00 16.00 16.00
0.08 39.50 0.00 39.50 20.50 4.10
dm-22 0.00 0.00 0.00 5.00 0.00 36.00 14.40
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.09 17.20 0.00 17.20 5.40 2.70
dm-25 0.00 0.00 0.00 6.00 0.00 48.00 16.00
0.10 16.00 0.00 16.00 5.50 3.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 48.00 19.20
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 5.00 0.00 44.00 17.60
0.01 3.00 0.00 3.00 3.00 1.50
dm-16 0.00 0.00 0.00 3.00 0.00 32.00 21.33
0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 40.00 20.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 32.00 21.33
0.04 13.00 0.00 13.00 6.67 2.00
dm-19 0.00 0.00 0.00 3.00 0.00 28.00 18.67
0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 6.00 0.00 64.00 21.33
0.11 17.67 0.00 17.67 3.17 1.90
dm-21 0.00 0.00 0.00 1.00 0.00 12.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 1.00 0.00 12.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 3.00 0.00 20.00 13.33
0.00 0.33 0.00 0.33 0.33 0.10
dm-25 0.00 0.00 0.00 6.00 0.00 52.00 17.33
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 18.00 0.00 236.00 26.22
0.42 23.28 0.00 23.28 3.72 6.70
dm-15 0.00 0.00 0.00 15.00 0.00 240.00 32.00
0.12 8.27 0.00 8.27 2.00 3.00
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00
0.23 22.80 0.00 22.80 6.50 6.50
dm-17 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.06 9.83 0.00 9.83 5.00 3.00
dm-18 0.00 0.00 0.00 12.00 0.00 152.00 25.33
0.10 8.17 0.00 8.17 3.17 3.80
dm-19 0.00 0.00 0.00 12.00 0.00 136.00 22.67
0.14 11.58 0.00 11.58 3.42 4.10
dm-20 0.00 0.00 0.00 19.00 0.00 296.00 31.16
0.20 10.63 0.00 10.63 1.84 3.50
dm-21 0.00 0.00 0.00 14.00 0.00 132.00 18.86
0.17 12.14 0.00 12.14 3.71 5.20
dm-22 0.00 0.00 0.00 10.00 0.00 84.00 16.80
0.12 12.00 0.00 12.00 6.60 6.60
dm-24 0.00 0.00 0.00 5.00 0.00 40.00 16.00
0.10 19.40 0.00 19.40 9.80 4.90
dm-25 0.00 0.00 0.00 9.00 0.00 152.00 33.78
0.08 9.33 0.00 9.33 3.44 3.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 112.00 28.00
0.08 9.75 0.00 9.75 3.88 3.10
dm-15 0.00 0.00 0.00 14.00 0.00 156.00 22.29
0.13 9.43 0.00 9.43 2.71 3.80
dm-16 0.00 0.00 0.00 12.00 0.00 132.00 22.00
0.11 8.83 0.00 8.83 2.17 2.60
dm-17 0.00 0.00 0.00 8.00 0.00 116.00 29.00
0.08 10.12 0.00 10.12 4.25 3.40
dm-18 0.00 0.00 0.00 8.00 0.00 100.00 25.00
0.08 9.62 0.00 9.62 5.00 4.00
dm-19 0.00 0.00 0.00 20.00 0.00 212.00 21.20
0.25 12.75 0.00 12.75 2.60 5.20
dm-20 0.00 0.00 0.00 16.00 0.00 168.00 21.00
0.04 2.69 0.00 2.69 0.81 1.30
dm-21 0.00 0.00 0.00 5.00 0.00 48.00 19.20
0.05 10.80 0.00 10.80 7.80 3.90
dm-22 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.03 6.80 0.00 6.80 3.40 1.70
dm-24 0.00 0.00 0.00 8.00 0.00 60.00 15.00
0.09 11.75 0.00 11.75 2.25 1.80
dm-25 0.00 0.00 0.00 12.00 0.00 160.00 26.67
0.06 5.08 0.00 5.08 1.25 1.50
On 11/17/2018 10:19 PM, Alex Litvak wrote:
I stand corrected, I looked at the device iostat, but it was partitioned. Here
is a more correct picture of what is going on now.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 19.00 0.00 4116.00 433.26
0.01 0.68 0.00 0.68 0.05 0.10
dm-15 0.00 0.00 0.00 35.00 0.00 8224.00 469.94
0.03 0.86 0.00 0.86 0.06 0.20
dm-16 0.00 0.00 0.00 53.00 0.00 12428.00 468.98
0.11 2.04 0.00 2.04 0.17 0.90
dm-17 0.00 0.00 0.00 43.00 0.00 8344.00 388.09
0.09 2.14 0.00 2.14 0.42 1.80
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 75.00 0.00 16824.00 448.64
0.08 1.11 0.00 1.11 0.08 0.60
dm-20 0.00 0.00 0.00 70.00 0.00 16452.00 470.06
0.06 0.90 0.00 0.90 0.09 0.60
dm-21 0.00 0.00 0.00 18.00 0.00 4112.00 456.89
0.02 1.00 0.00 1.00 0.11 0.20
dm-22 0.00 0.00 0.00 53.00 0.00 12324.00 465.06
0.06 0.70 0.00 0.70 0.08 0.40
dm-24 0.00 0.00 0.00 18.00 0.00 4272.00 474.67
0.02 1.06 0.00 1.06 0.17 0.30
dm-25 0.00 0.00 0.00 74.00 0.00 16916.00 457.19
0.09 1.26 0.00 1.26 0.18 1.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 17.00 0.00 4108.00 483.29
0.02 1.00 0.00 1.00 0.06 0.10
dm-16 0.00 0.00 0.00 34.00 0.00 8208.00 482.82
0.03 1.00 0.00 1.00 0.06 0.20
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 36.00 0.00 8220.00 456.67
0.05 1.33 0.00 1.33 0.08 0.30
dm-19 0.00 0.00 0.00 1.00 0.00 8.00 16.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 36.00 0.00 8288.00 460.44
0.05 1.42 0.00 1.42 0.08 0.30
dm-21 0.00 0.00 0.00 34.00 0.00 8208.00 482.82
0.03 1.00 0.00 1.00 0.06 0.20
dm-22 0.00 0.00 0.00 18.00 0.00 4128.00 458.67
0.04 3.22 0.00 3.22 0.17 0.30
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 20.00 0.00 4032.00 403.20
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 1.00 0.00 20.00 40.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 28.00 14.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 36.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 2.00 0.00 20.00 20.00
0.01 2.50 0.00 2.50 2.50 0.50
dm-20 0.00 0.00 0.00 6.00 0.00 96.00 32.00
0.02 3.33 0.00 3.33 2.00 1.20
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 2.00 0.00 32.00 32.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 22.00 0.00 4184.00 380.36
0.10 4.59 0.00 4.59 0.95 2.10
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 1928.00 482.00
0.01 1.00 0.00 1.00 0.12 0.10
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 3.00 0.00 312.00 208.00
0.00 0.33 0.00 0.33 0.33 0.10
dm-17 0.00 0.00 0.00 18.00 0.00 4264.00 473.78
0.03 1.67 0.00 1.67 0.11 0.20
dm-18 0.00 0.00 0.00 17.00 0.00 4104.00 482.82
0.03 1.82 0.00 1.82 0.12 0.20
dm-19 0.00 0.00 0.00 18.00 0.00 4112.00 456.89
0.02 1.06 0.00 1.06 0.11 0.20
dm-20 0.00 0.00 0.00 32.00 0.00 4308.00 269.25
0.03 0.81 0.00 0.81 0.34 1.10
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 8.00 0.00 540.00 135.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 35.00 0.00 8228.00 470.17
0.03 0.97 0.00 0.97 0.06 0.20
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
On 11/17/2018 5:27 PM, John Petrini wrote:
The iostat isn't very helpful because there are not many writes. I'd recommend
disabling cstates entirely, not sure it's your problem but it's good practice
and if your cluster goes as idle as your
iostat suggests it could be the culprit.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
