Output of ceph status:
cluster 18e87fd8-17c1-4045-a1a2-07aac106f200
health HEALTH_WARN
1 pgs backfill_wait
56 pgs degraded
1 pgs recovering
55 pgs recovery_wait
56 pgs stuck degraded
57 pgs stuck unclean
recovery 50570/1369003 objects degraded (3.694%)
recovery 854/1369003 objects misplaced (0.062%)
monmap e2: 1 mons at {ip-172-31-16-102=172.31.16.102:6789/0}
election epoch 4, quorum 0 ip-172-31-16-102
mgr active: ip-172-31-16-102
osdmap e247: 32 osds: 32 up, 31 in; 1 remapped pgs
flags sortbitwise,require_jewel_osds,require_kraken_osds
pgmap v10860: 256 pgs, 1 pools, 1975 GB data, 111 kobjects
2923 GB used, 6836 GB / 9760 GB avail
50570/1369003 objects degraded (3.694%)
854/1369003 objects misplaced (0.062%)
199 active+clean
55 active+recovery_wait+degraded
1 active+remapped+backfill_wait
1 active+recovering+degraded
client io 513 MB/s rd, 131 op/s rd, 0 op/s wr
Output of ceph osd tree:
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 9.83984 root default
-2 0.30750 host ip-172-31-24-96
0 0.30750 osd.0 up 1.00000 1.00000
-3 0.30750 host ip-172-31-30-32
1 0.30750 osd.1 up 1.00000 1.00000
-4 0.30750 host ip-172-31-28-36
2 0.30750 osd.2 up 1.00000 1.00000
-5 0.30750 host ip-172-31-18-100
3 0.30750 osd.3 up 1.00000 1.00000
-6 0.30750 host ip-172-31-25-240
4 0.30750 osd.4 up 1.00000 1.00000
-7 0.30750 host ip-172-31-24-110
5 0.30750 osd.5 up 1.00000 1.00000
-8 0.30750 host ip-172-31-20-245
6 0.30750 osd.6 up 0 1.00000
-9 0.30750 host ip-172-31-17-241
7 0.30750 osd.7 up 1.00000 1.00000
-10 0.30750 host ip-172-31-18-107
8 0.30750 osd.8 up 1.00000 1.00000
-11 0.30750 host ip-172-31-21-170
9 0.30750 osd.9 up 1.00000 1.00000
-12 0.30750 host ip-172-31-21-29
10 0.30750 osd.10 up 1.00000 1.00000
-13 0.30750 host ip-172-31-23-220
11 0.30750 osd.11 up 1.00000 1.00000
-14 0.30750 host ip-172-31-24-154
12 0.30750 osd.12 up 1.00000 1.00000
-15 0.30750 host ip-172-31-26-25
13 0.30750 osd.13 up 1.00000 1.00000
-16 0.30750 host ip-172-31-20-28
14 0.30750 osd.14 up 1.00000 1.00000
-17 0.30750 host ip-172-31-23-90
15 0.30750 osd.15 up 1.00000 1.00000
-18 0.30750 host ip-172-31-31-197
16 0.30750 osd.16 up 1.00000 1.00000
-19 0.30750 host ip-172-31-29-195
17 0.30750 osd.17 up 1.00000 1.00000
-20 0.30750 host ip-172-31-28-9
18 0.30750 osd.18 up 1.00000 1.00000
-21 0.30750 host ip-172-31-25-199
19 0.30750 osd.19 up 1.00000 1.00000
-22 0.30750 host ip-172-31-25-187
20 0.30750 osd.20 up 1.00000 1.00000
-23 0.30750 host ip-172-31-31-57
21 0.30750 osd.21 up 1.00000 1.00000
-24 0.30750 host ip-172-31-20-64
22 0.30750 osd.22 up 1.00000 1.00000
-25 0.30750 host ip-172-31-26-255
23 0.30750 osd.23 up 1.00000 1.00000
-26 0.30750 host ip-172-31-18-146
24 0.30750 osd.24 up 1.00000 1.00000
-27 0.30750 host ip-172-31-22-16
25 0.30750 osd.25 up 1.00000 1.00000
-28 0.30750 host ip-172-31-26-152
26 0.30750 osd.26 up 1.00000 1.00000
-29 0.30750 host ip-172-31-24-215
27 0.30750 osd.27 up 1.00000 1.00000
-30 0.30750 host ip-172-31-24-138
28 0.30750 osd.28 up 1.00000 1.00000
-31 0.30750 host ip-172-31-24-10
29 0.30750 osd.29 up 1.00000 1.00000
-32 0.30750 host ip-172-31-20-79
30 0.30750 osd.30 up 1.00000 1.00000
-33 0.30750 host ip-172-31-23-140
31 0.30750 osd.31 up 1.00000 1.00000
Output of ceph health detail:
HEALTH_WARN 1 pgs backfill_wait; 55 pgs degraded; 1 pgs recovering; 54 pgs
recovery_wait; 55 pgs stuck degraded; 56 pgs stuck unclean; recovery
49688/1369003 objects degraded (3.630%); recovery 854/1369003 objects misplaced
(0.062%)
pg 3.b is stuck unclean for 3620.478034, current state
active+recovery_wait+degraded, last acting [30,11,31,20,17,3,1,25,28,29,7,24]
pg 3.f is stuck unclean for 2574.807568, current state
active+recovery_wait+degraded, last acting [27,26,3,0,18,19,11,10,9,17,8,21]
pg 3.11 is stuck unclean for 5031.004347, current state
active+recovery_wait+degraded, last acting [30,3,2,7,4,17,14,23,5,16,13,29]
pg 3.24 is stuck unclean for 3611.733994, current state
active+recovery_wait+degraded, last acting [12,14,17,30,1,16,24,3,27,22,0,18]
pg 3.2f is stuck unclean for 5562.733823, current state
active+recovery_wait+degraded, last acting [22,24,5,27,10,2,3,0,17,15,23,7]
pg 3.3b is stuck unclean for 5000.158982, current state
active+recovery_wait+degraded, last acting [19,11,16,4,28,30,21,8,31,5,13,27]
pg 3.aa is stuck unclean for 4827.355024, current state
active+recovery_wait+degraded, last acting [12,10,20,0,22,9,19,24,2,3,16,5]
pg 3.79 is stuck unclean for 3652.909790, current state
active+remapped+backfill_wait, last acting [25,1,4,19,23,6,5,2,27,12,16,8]
pg 3.ab is stuck unclean for 5607.537767, current state
active+recovery_wait+degraded, last acting [19,3,30,11,0,4,22,25,16,12,8,14]
pg 3.1b is stuck unclean for 4704.402285, current state
active+recovery_wait+degraded, last acting [17,10,7,27,16,26,23,1,11,9,0,14]
pg 3.7a is stuck unclean for 4465.053715, current state
active+recovery_wait+degraded, last acting [17,8,2,19,14,16,31,20,29,26,15,9]
pg 3.49 is stuck unclean for 4052.718824, current state
active+recovery_wait+degraded, last acting [9,3,1,16,8,7,11,14,19,13,12,18]
pg 3.ac is stuck unclean for 4940.338938, current state
active+recovery_wait+degraded, last acting [8,2,3,0,5,10,18,12,16,7,17,1]
pg 3.83 is stuck unclean for 4381.695898, current state
active+recovery_wait+degraded, last acting [18,12,21,15,16,27,3,26,28,5,20,19]
pg 3.52 is stuck unclean for 4337.289527, current state
active+recovery_wait+degraded, last acting [17,24,20,23,4,14,18,27,8,22,9,31]
pg 3.ae is stuck unclean for 5107.221614, current state
active+recovery_wait+degraded, last acting [27,3,25,7,11,9,8,30,13,23,0,2]
pg 3.b4 is stuck unclean for 4687.534444, current state
active+recovery_wait+degraded, last acting [7,30,10,19,31,23,13,1,18,25,28,0]
pg 3.bd is stuck unclean for 5099.627501, current state
active+recovery_wait+degraded, last acting [4,3,15,7,23,17,9,31,20,12,21,24]
pg 3.ad is stuck unclean for 4907.243126, current state
active+recovery_wait+degraded, last acting [24,8,1,21,30,27,25,13,7,0,11,19]
pg 3.af is stuck unclean for 3950.747953, current state
active+recovery_wait+degraded, last acting [30,23,25,8,11,15,13,14,18,24,0,21]
pg 3.7e is stuck unclean for 3461.008617, current state
active+recovery_wait+degraded, last acting [22,15,12,16,4,0,30,14,31,23,10,17]
pg 3.97 is stuck unclean for 5330.236878, current state
active+recovery_wait+degraded, last acting [19,10,24,14,13,21,2,8,31,29,30,20]
pg 3.f2 is stuck unclean for 2731.659626, current state
active+recovery_wait+degraded, last acting [31,13,2,16,0,14,3,29,1,26,7,10]
pg 3.c6 is stuck unclean for 6306.423348, current state
active+recovery_wait+degraded, last acting [0,10,28,31,12,4,5,25,24,13,2,18]
pg 3.67 is stuck unclean for 5118.168893, current state
active+recovery_wait+degraded, last acting [9,15,29,18,25,7,17,30,4,12,26,23]
pg 3.b5 is stuck unclean for 4369.784919, current state
active+recovery_wait+degraded, last acting [17,3,28,5,15,4,16,25,11,0,26,31]
pg 3.5b is stuck unclean for 2621.626018, current state
active+recovery_wait+degraded, last acting [4,15,14,30,28,1,12,10,2,29,24,18]
pg 3.f8 is stuck unclean for 4522.911060, current state
active+recovery_wait+degraded, last acting [18,3,2,29,26,9,17,5,22,13,31,21]
pg 3.3c is stuck unclean for 3337.005364, current state
active+recovery_wait+degraded, last acting [2,24,16,20,18,25,26,10,23,12,19,31]
pg 3.ec is stuck unclean for 5592.096810, current state
active+recovery_wait+degraded, last acting [18,11,27,9,26,1,22,31,0,10,24,15]
pg 3.92 is stuck unclean for 5533.331735, current state
active+recovery_wait+degraded, last acting [13,11,31,0,22,12,9,10,28,3,21,2]
pg 3.c0 is stuck unclean for 5214.160745, current state
active+recovery_wait+degraded, last acting [12,28,31,30,2,24,15,22,9,18,3,25]
pg 3.61 is stuck unclean for 3880.126824, current state
active+recovery_wait+degraded, last acting [4,8,24,15,0,28,16,19,13,1,18,27]
pg 3.eb is stuck unclean for 5268.977639, current state
active+recovery_wait+degraded, last acting [8,25,1,16,30,14,18,9,21,24,4,7]
pg 3.b8 is stuck unclean for 4399.307382, current state
active+recovery_wait+degraded, last acting [16,21,28,30,7,17,1,2,14,8,0,13]
pg 3.d1 is stuck unclean for 3577.663496, current state
active+recovery_wait+degraded, last acting [0,28,22,31,20,4,11,10,2,1,25,24]
pg 3.89 is stuck unclean for 5730.882619, current state
active+recovery_wait+degraded, last acting [4,5,1,2,0,9,24,11,14,13,15,28]
pg 3.e8 is stuck unclean for 6516.175205, current state
active+recovery_wait+degraded, last acting [23,11,28,7,8,14,27,9,30,31,24,5]
pg 3.34 is stuck unclean for 5472.972458, current state
active+recovery_wait+degraded, last acting [19,8,16,31,27,22,18,0,30,4,1,11]
pg 3.fa is stuck unclean for 3740.578030, current state
active+recovering+degraded, last acting [2,1,18,17,25,19,23,24,3,8,12,30]
pg 3.9b is stuck unclean for 4914.758904, current state
active+recovery_wait+degraded, last acting [23,15,26,27,12,28,3,4,8,30,0,1]
pg 3.91 is stuck unclean for 4486.518498, current state
active+recovery_wait+degraded, last acting [9,14,27,11,1,23,7,17,5,16,18,0]
pg 3.26 is stuck unclean for 5068.577531, current state
active+recovery_wait+degraded, last acting [9,25,16,24,12,2,17,22,31,29,26,7]
pg 3.85 is stuck unclean for 5229.745995, current state
active+recovery_wait+degraded, last acting [12,3,22,30,16,2,20,28,8,0,25,4]
pg 3.e0 is stuck unclean for 2662.946214, current state
active+recovery_wait+degraded, last acting [17,28,20,19,4,9,15,25,16,0,23,22]
pg 3.81 is stuck unclean for 4265.267581, current state
active+recovery_wait+degraded, last acting [13,4,0,19,10,17,16,24,12,26,3,20]
pg 3.14 is stuck unclean for 4366.392617, current state
active+recovery_wait+degraded, last acting [20,0,18,16,30,25,12,31,4,3,5,29]
pg 3.7b is stuck unclean for 5133.369388, current state
active+recovery_wait+degraded, last acting [4,22,30,1,21,5,12,19,17,0,2,23]
pg 3.78 is stuck unclean for 5286.596260, current state
active+recovery_wait+degraded, last acting [23,26,18,3,13,14,19,7,0,12,25,17]
pg 3.d0 is stuck unclean for 5293.763984, current state
active+recovery_wait+degraded, last acting [24,10,4,12,2,25,9,23,8,15,29,7]
pg 3.71 is stuck unclean for 4571.041709, current state
active+recovery_wait+degraded, last acting [25,9,27,29,16,13,11,3,18,19,26,4]
pg 3.ca is stuck unclean for 5465.875924, current state
active+recovery_wait+degraded, last acting [17,24,21,3,16,14,25,10,2,5,28,18]
pg 3.6b is stuck unclean for 4627.831337, current state
active+recovery_wait+degraded, last acting [21,1,4,20,27,7,17,24,3,0,29,25]
pg 3.69 is stuck unclean for 4757.583113, current state
active+recovery_wait+degraded, last acting [8,26,4,30,11,9,16,12,18,27,14,19]
pg 3.5c is stuck unclean for 5362.827077, current state
active+recovery_wait+degraded, last acting [14,29,4,1,19,17,9,0,3,16,24,2]
pg 3.51 is stuck unclean for 2778.350320, current state
active+recovery_wait+degraded, last acting [13,31,11,22,25,30,1,3,27,23,21,17]
pg 3.b is stuck degraded for 2292.794292, current state
active+recovery_wait+degraded, last acting [30,11,31,20,17,3,1,25,28,29,7,24]
pg 3.f is stuck degraded for 2292.771080, current state
active+recovery_wait+degraded, last acting [27,26,3,0,18,19,11,10,9,17,8,21]
pg 3.11 is stuck degraded for 2292.797135, current state
active+recovery_wait+degraded, last acting [30,3,2,7,4,17,14,23,5,16,13,29]
pg 3.24 is stuck degraded for 2292.825615, current state
active+recovery_wait+degraded, last acting [12,14,17,30,1,16,24,3,27,22,0,18]
pg 3.2f is stuck degraded for 2292.787887, current state
active+recovery_wait+degraded, last acting [22,24,5,27,10,2,3,0,17,15,23,7]
pg 3.3b is stuck degraded for 2292.823674, current state
active+recovery_wait+degraded, last acting [19,11,16,4,28,30,21,8,31,5,13,27]
pg 3.3c is stuck degraded for 2292.813364, current state
active+recovery_wait+degraded, last acting [2,24,16,20,18,25,26,10,23,12,19,31]
pg 3.49 is stuck degraded for 2292.804643, current state
active+recovery_wait+degraded, last acting [9,3,1,16,8,7,11,14,19,13,12,18]
pg 3.51 is stuck degraded for 2292.798396, current state
active+recovery_wait+degraded, last acting [13,31,11,22,25,30,1,3,27,23,21,17]
pg 3.52 is stuck degraded for 2292.799715, current state
active+recovery_wait+degraded, last acting [17,24,20,23,4,14,18,27,8,22,9,31]
pg 3.5b is stuck degraded for 2292.776512, current state
active+recovery_wait+degraded, last acting [4,15,14,30,28,1,12,10,2,29,24,18]
pg 3.5c is stuck degraded for 2292.808334, current state
active+recovery_wait+degraded, last acting [14,29,4,1,19,17,9,0,3,16,24,2]
pg 3.69 is stuck degraded for 2292.809014, current state
active+recovery_wait+degraded, last acting [8,26,4,30,11,9,16,12,18,27,14,19]
pg 3.78 is stuck degraded for 2292.798826, current state
active+recovery_wait+degraded, last acting [23,26,18,3,13,14,19,7,0,12,25,17]
pg 3.1b is stuck degraded for 2292.798541, current state
active+recovery_wait+degraded, last acting [17,10,7,27,16,26,23,1,11,9,0,14]
pg 3.7a is stuck degraded for 2292.803093, current state
active+recovery_wait+degraded, last acting [17,8,2,19,14,16,31,20,29,26,15,9]
pg 3.14 is stuck degraded for 2292.793869, current state
active+recovery_wait+degraded, last acting [20,0,18,16,30,25,12,31,4,3,5,29]
pg 3.7b is stuck degraded for 2292.782484, current state
active+recovery_wait+degraded, last acting [4,22,30,1,21,5,12,19,17,0,2,23]
pg 3.7e is stuck degraded for 2292.774470, current state
active+recovery_wait+degraded, last acting [22,15,12,16,4,0,30,14,31,23,10,17]
pg 3.83 is stuck degraded for 2292.795022, current state
active+recovery_wait+degraded, last acting [18,12,21,15,16,27,3,26,28,5,20,19]
pg 3.26 is stuck degraded for 2292.807846, current state
active+recovery_wait+degraded, last acting [9,25,16,24,12,2,17,22,31,29,26,7]
pg 3.85 is stuck degraded for 2292.813155, current state
active+recovery_wait+degraded, last acting [12,3,22,30,16,2,20,28,8,0,25,4]
pg 3.91 is stuck degraded for 2292.810660, current state
active+recovery_wait+degraded, last acting [9,14,27,11,1,23,7,17,5,16,18,0]
pg 3.92 is stuck degraded for 2292.809843, current state
active+recovery_wait+degraded, last acting [13,11,31,0,22,12,9,10,28,3,21,2]
pg 3.97 is stuck degraded for 2292.782984, current state
active+recovery_wait+degraded, last acting [19,10,24,14,13,21,2,8,31,29,30,20]
pg 3.aa is stuck degraded for 2292.805640, current state
active+recovery_wait+degraded, last acting [12,10,20,0,22,9,19,24,2,3,16,5]
pg 3.ab is stuck degraded for 2292.766750, current state
active+recovery_wait+degraded, last acting [19,3,30,11,0,4,22,25,16,12,8,14]
pg 3.ac is stuck degraded for 2292.817247, current state
active+recovery_wait+degraded, last acting [8,2,3,0,5,10,18,12,16,7,17,1]
pg 3.ad is stuck degraded for 2292.811631, current state
active+recovery_wait+degraded, last acting [24,8,1,21,30,27,25,13,7,0,11,19]
pg 3.ae is stuck degraded for 2292.765243, current state
active+recovery_wait+degraded, last acting [27,3,25,7,11,9,8,30,13,23,0,2]
pg 3.af is stuck degraded for 2292.785730, current state
active+recovery_wait+degraded, last acting [30,23,25,8,11,15,13,14,18,24,0,21]
pg 3.b4 is stuck degraded for 2292.807764, current state
active+recovery_wait+degraded, last acting [7,30,10,19,31,23,13,1,18,25,28,0]
pg 3.b5 is stuck degraded for 2292.802932, current state
active+recovery_wait+degraded, last acting [17,3,28,5,15,4,16,25,11,0,26,31]
pg 3.b8 is stuck degraded for 2292.789546, current state
active+recovery_wait+degraded, last acting [16,21,28,30,7,17,1,2,14,8,0,13]
pg 3.bd is stuck degraded for 2292.777194, current state
active+recovery_wait+degraded, last acting [4,3,15,7,23,17,9,31,20,12,21,24]
pg 3.61 is stuck degraded for 2292.780051, current state
active+recovery_wait+degraded, last acting [4,8,24,15,0,28,16,19,13,1,18,27]
pg 3.c0 is stuck degraded for 2292.813792, current state
active+recovery_wait+degraded, last acting [12,28,31,30,2,24,15,22,9,18,3,25]
pg 3.67 is stuck degraded for 2292.810551, current state
active+recovery_wait+degraded, last acting [9,15,29,18,25,7,17,30,4,12,26,23]
pg 3.c6 is stuck degraded for 2292.813695, current state
active+recovery_wait+degraded, last acting [0,10,28,31,12,4,5,25,24,13,2,18]
pg 3.6b is stuck degraded for 2292.784572, current state
active+recovery_wait+degraded, last acting [21,1,4,20,27,7,17,24,3,0,29,25]
pg 3.ca is stuck degraded for 2292.802657, current state
active+recovery_wait+degraded, last acting [17,24,21,3,16,14,25,10,2,5,28,18]
pg 3.71 is stuck degraded for 2292.745595, current state
active+recovery_wait+degraded, last acting [25,9,27,29,16,13,11,3,18,19,26,4]
pg 3.d0 is stuck degraded for 2292.810869, current state
active+recovery_wait+degraded, last acting [24,10,4,12,2,25,9,23,8,15,29,7]
pg 3.d1 is stuck degraded for 2292.797445, current state
active+recovery_wait+degraded, last acting [0,28,22,31,20,4,11,10,2,1,25,24]
pg 3.81 is stuck degraded for 2292.803404, current state
active+recovery_wait+degraded, last acting [13,4,0,19,10,17,16,24,12,26,3,20]
pg 3.e0 is stuck degraded for 2292.763504, current state
active+recovery_wait+degraded, last acting [17,28,20,19,4,9,15,25,16,0,23,22]
pg 3.89 is stuck degraded for 2292.779741, current state
active+recovery_wait+degraded, last acting [4,5,1,2,0,9,24,11,14,13,15,28]
pg 3.e8 is stuck degraded for 2292.762470, current state
active+recovery_wait+degraded, last acting [23,11,28,7,8,14,27,9,30,31,24,5]
pg 3.eb is stuck degraded for 2292.823880, current state
active+recovery_wait+degraded, last acting [8,25,1,16,30,14,18,9,21,24,4,7]
pg 3.ec is stuck degraded for 2292.820337, current state
active+recovery_wait+degraded, last acting [18,11,27,9,26,1,22,31,0,10,24,15]
pg 3.f2 is stuck degraded for 2292.781004, current state
active+recovery_wait+degraded, last acting [31,13,2,16,0,14,3,29,1,26,7,10]
pg 3.f8 is stuck degraded for 2292.791270, current state
active+recovery_wait+degraded, last acting [18,3,2,29,26,9,17,5,22,13,31,21]
pg 3.34 is stuck degraded for 2292.824069, current state
active+recovery_wait+degraded, last acting [19,8,16,31,27,22,18,0,30,4,1,11]
pg 3.9b is stuck degraded for 2292.762747, current state
active+recovery_wait+degraded, last acting [23,15,26,27,12,28,3,4,8,30,0,1]
pg 3.fa is stuck degraded for 2292.808553, current state
active+recovering+degraded, last acting [2,1,18,17,25,19,23,24,3,8,12,30]
pg 3.fa is active+recovering+degraded, acting [2,1,18,17,25,19,23,24,3,8,12,30]
pg 3.f8 is active+recovery_wait+degraded, acting
[18,3,2,29,26,9,17,5,22,13,31,21]
pg 3.f2 is active+recovery_wait+degraded, acting
[31,13,2,16,0,14,3,29,1,26,7,10]
pg 3.ec is active+recovery_wait+degraded, acting
[18,11,27,9,26,1,22,31,0,10,24,15]
pg 3.eb is active+recovery_wait+degraded, acting
[8,25,1,16,30,14,18,9,21,24,4,7]
pg 3.e8 is active+recovery_wait+degraded, acting
[23,11,28,7,8,14,27,9,30,31,24,5]
pg 3.e0 is active+recovery_wait+degraded, acting
[17,28,20,19,4,9,15,25,16,0,23,22]
pg 3.d1 is active+recovery_wait+degraded, acting
[0,28,22,31,20,4,11,10,2,1,25,24]
pg 3.d0 is active+recovery_wait+degraded, acting
[24,10,4,12,2,25,9,23,8,15,29,7]
pg 3.ca is active+recovery_wait+degraded, acting
[17,24,21,3,16,14,25,10,2,5,28,18]
pg 3.c6 is active+recovery_wait+degraded, acting
[0,10,28,31,12,4,5,25,24,13,2,18]
pg 3.c0 is active+recovery_wait+degraded, acting
[12,28,31,30,2,24,15,22,9,18,3,25]
pg 3.bd is active+recovery_wait+degraded, acting
[4,3,15,7,23,17,9,31,20,12,21,24]
pg 3.b8 is active+recovery_wait+degraded, acting
[16,21,28,30,7,17,1,2,14,8,0,13]
pg 3.b5 is active+recovery_wait+degraded, acting
[17,3,28,5,15,4,16,25,11,0,26,31]
pg 3.b4 is active+recovery_wait+degraded, acting
[7,30,10,19,31,23,13,1,18,25,28,0]
pg 3.af is active+recovery_wait+degraded, acting
[30,23,25,8,11,15,13,14,18,24,0,21]
pg 3.ae is active+recovery_wait+degraded, acting [27,3,25,7,11,9,8,30,13,23,0,2]
pg 3.ad is active+recovery_wait+degraded, acting
[24,8,1,21,30,27,25,13,7,0,11,19]
pg 3.ac is active+recovery_wait+degraded, acting [8,2,3,0,5,10,18,12,16,7,17,1]
pg 3.ab is active+recovery_wait+degraded, acting
[19,3,30,11,0,4,22,25,16,12,8,14]
pg 3.aa is active+recovery_wait+degraded, acting
[12,10,20,0,22,9,19,24,2,3,16,5]
pg 3.9b is active+recovery_wait+degraded, acting
[23,15,26,27,12,28,3,4,8,30,0,1]
pg 3.97 is active+recovery_wait+degraded, acting
[19,10,24,14,13,21,2,8,31,29,30,20]
pg 3.92 is active+recovery_wait+degraded, acting
[13,11,31,0,22,12,9,10,28,3,21,2]
pg 3.91 is active+recovery_wait+degraded, acting
[9,14,27,11,1,23,7,17,5,16,18,0]
pg 3.89 is active+recovery_wait+degraded, acting [4,5,1,2,0,9,24,11,14,13,15,28]
pg 3.85 is active+recovery_wait+degraded, acting
[12,3,22,30,16,2,20,28,8,0,25,4]
pg 3.83 is active+recovery_wait+degraded, acting
[18,12,21,15,16,27,3,26,28,5,20,19]
pg 3.81 is active+recovery_wait+degraded, acting
[13,4,0,19,10,17,16,24,12,26,3,20]
pg 3.7e is active+recovery_wait+degraded, acting
[22,15,12,16,4,0,30,14,31,23,10,17]
pg 3.7b is active+recovery_wait+degraded, acting
[4,22,30,1,21,5,12,19,17,0,2,23]
pg 3.7a is active+recovery_wait+degraded, acting
[17,8,2,19,14,16,31,20,29,26,15,9]
pg 3.79 is active+remapped+backfill_wait, acting [25,1,4,19,23,6,5,2,27,12,16,8]
pg 3.78 is active+recovery_wait+degraded, acting
[23,26,18,3,13,14,19,7,0,12,25,17]
pg 3.71 is active+recovery_wait+degraded, acting
[25,9,27,29,16,13,11,3,18,19,26,4]
pg 3.6b is active+recovery_wait+degraded, acting
[21,1,4,20,27,7,17,24,3,0,29,25]
pg 3.69 is active+recovery_wait+degraded, acting
[8,26,4,30,11,9,16,12,18,27,14,19]
pg 3.67 is active+recovery_wait+degraded, acting
[9,15,29,18,25,7,17,30,4,12,26,23]
pg 3.61 is active+recovery_wait+degraded, acting
[4,8,24,15,0,28,16,19,13,1,18,27]
pg 3.5c is active+recovery_wait+degraded, acting [14,29,4,1,19,17,9,0,3,16,24,2]
pg 3.5b is active+recovery_wait+degraded, acting
[4,15,14,30,28,1,12,10,2,29,24,18]
pg 3.52 is active+recovery_wait+degraded, acting
[17,24,20,23,4,14,18,27,8,22,9,31]
pg 3.51 is active+recovery_wait+degraded, acting
[13,31,11,22,25,30,1,3,27,23,21,17]
pg 3.49 is active+recovery_wait+degraded, acting
[9,3,1,16,8,7,11,14,19,13,12,18]
pg 3.3c is active+recovery_wait+degraded, acting
[2,24,16,20,18,25,26,10,23,12,19,31]
pg 3.3b is active+recovery_wait+degraded, acting
[19,11,16,4,28,30,21,8,31,5,13,27]
pg 3.34 is active+recovery_wait+degraded, acting
[19,8,16,31,27,22,18,0,30,4,1,11]
pg 3.2f is active+recovery_wait+degraded, acting
[22,24,5,27,10,2,3,0,17,15,23,7]
pg 3.26 is active+recovery_wait+degraded, acting
[9,25,16,24,12,2,17,22,31,29,26,7]
pg 3.24 is active+recovery_wait+degraded, acting
[12,14,17,30,1,16,24,3,27,22,0,18]
pg 3.1b is active+recovery_wait+degraded, acting
[17,10,7,27,16,26,23,1,11,9,0,14]
pg 3.14 is active+recovery_wait+degraded, acting
[20,0,18,16,30,25,12,31,4,3,5,29]
pg 3.11 is active+recovery_wait+degraded, acting
[30,3,2,7,4,17,14,23,5,16,13,29]
pg 3.f is active+recovery_wait+degraded, acting
[27,26,3,0,18,19,11,10,9,17,8,21]
pg 3.b is active+recovery_wait+degraded, acting
[30,11,31,20,17,3,1,25,28,29,7,24]
recovery 49688/1369003 objects degraded (3.630%)
recovery 854/1369003 objects misplaced (0.062%)
> Am 19.09.2017 um 22:15 schrieb David Turner <[email protected]>:
>
> Can you please provide the output of `ceph status`, `ceph osd tree`, and
> `ceph health detail`? Thank you.
>
> On Tue, Sep 19, 2017 at 2:59 PM Jonas Jaszkowic
> <[email protected] <mailto:[email protected]>> wrote:
> Hi all,
>
> I have setup a Ceph cluster consisting of one monitor, 32 OSD hosts (1 OSD of
> size 320GB per host) and 16 clients which are reading
> and writing to the cluster. I have one erasure coded pool (shec plugin) with
> k=8, m=4, c=3 and pg_num=256. Failure domain is host.
> I am able to reach a HEALTH_OK state and everything is working as expected.
> The pool was populated with
> 114048 files of different sizes ranging from 1kB to 4GB. Total amount of data
> in the pool was around 3TB. The capacity of the
> pool was around 10TB.
>
> I want to evaluate how Ceph is rebalancing data in case of an OSD loss while
> clients are still reading. To do so, I am killing one OSD on purpose
> via ceph osd out <osd-id> without adding a new one, i.e. I have 31 OSDs left.
> Ceph seems to notice this failure and starts to rebalance data
> which I can observe with the ceph -w command.
>
> However, Ceph failed to rebalance the data. The recovering process seemed to
> be stuck at a random point. I waited more than 12h but the
> number of degraded objects did not reduce and some PGs were stuck. Why is
> this happening? Based on the number of OSDs and the k,m,c values
> there should be enough hosts and OSDs to be able to recover from a single OSD
> failure?
>
> Thank you in advance!
> _______________________________________________
> ceph-users mailing list
> [email protected] <mailto:[email protected]>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
