Many operations in the OpenStack cluster are stuck because of this. For example, a VM cannot be removed because of operations stuck on osd.19:
2015-08-17 09:34:08.116274 7fa61e57a700 0 log_channel(cluster) log [WRN] : slow request 1920.261825 seconds old, received at 2015-08-17 09:02:07.853997: osd_op(client.4705573.0:4384 rbd_data.47a42a1fba00d3.000000000000110d [delete] 5.283a4ec7 ack+ondisk+write+known_if_redirected e61799) currently no flag points reached 2015-08-17 09:34:08.116279 7fa61e57a700 0 log_channel(cluster) log [WRN] : slow request 1920.157696 seconds old, received at 2015-08-17 09:02:07.958126: osd_op(client.4705573.0:4897 rbd_data.47a42a1fba00d3.000000000000130e [delete] 5.868caac7 ack+ondisk+write+known_if_redirected e61799) currently no flag points reached 2015-08-17 09:34:09.116537 7fa61e57a700 0 log_channel(cluster) log [WRN] : 38 slow requests, 9 included below; oldest blocked for > 68721.775549 secs 2015-08-17 09:34:09.116553 7fa61e57a700 0 log_channel(cluster) log [WRN] : slow request 1920.842824 seconds old, received at 2015-08-17 09:02:08.273620: osd_op(client.4705573.0:5846 rbd_data.47a42a1fba00d3.00000000000016c3 [delete] 5.dbd736c7 ack+ondisk+write+known_if_redirected e61799) currently no flag points reached rbd_data.47a42a1fba00d3.000000000000130e is an object in an VM disk that openstack is trying to delete. gr, Bart On Sun, Aug 16, 2015 at 1:27 PM Bart Vanbrabant <b...@vanbrabant.eu> wrote: > Hi, > > I have a ceph cluster with 26 osd's in 4 hosts only use for rbd for an > OpenStack cluster (started at 0.48 I think), currently running 0.94.2 on > Ubuntu 14.04. A few days ago one of the osd's was at 85% disk usage while > only 30% of the raw disk space is used. I ran reweight-by-utilization with > 150 was cutoff level. This reshuffled the data. I also noticed that the > number of pg was still at the level when there were less disks in the > cluster (1300). > > Based on the current guidelines I increased pg_num to 2048. It created the > placement groups except for the last one. To try to force the creation of > the pg I removed the OSD's (ceph osd out) assigned to that pg but that > makes no difference. Currently all OSD's are back in and two pg's are also > stuck in an unclean state: > > ceph health detail: > > HEALTH_WARN 2 pgs degraded; 2 pgs stale; 2 pgs stuck degraded; 1 pgs stuck > inactive; 2 pgs stuck stale; 3 pgs stuck unclean; 2 pgs stuck undersized; 2 > pgs undersized; 59 requests are blocked > 32 sec; 3 osds have slow > requests; recovery 221/549658 objects degraded (0.040%); recovery > 221/549658 objects misplaced (0.040%); pool volumes pg_num 2048 > pgp_num > 1400 > pg 5.6c7 is stuck inactive since forever, current state creating, last > acting [19,25] > pg 5.6c7 is stuck unclean since forever, current state creating, last > acting [19,25] > pg 5.2c7 is stuck unclean for 313513.609864, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 15.2bd is stuck unclean for 313513.610368, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 5.2c7 is stuck undersized for 308381.750768, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 15.2bd is stuck undersized for 308381.751913, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 5.2c7 is stuck degraded for 308381.750876, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 15.2bd is stuck degraded for 308381.752021, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 5.2c7 is stuck stale for 281750.295301, current state > stale+active+undersized+degraded+remapped, last acting [9] > pg 15.2bd is stuck stale for 281750.295293, current state > stale+active+undersized+degraded+remapped, last acting [9] > 16 ops are blocked > 268435 sec > 10 ops are blocked > 134218 sec > 10 ops are blocked > 1048.58 sec > 23 ops are blocked > 524.288 sec > 16 ops are blocked > 268435 sec on osd.1 > 8 ops are blocked > 134218 sec on osd.17 > 2 ops are blocked > 134218 sec on osd.19 > 10 ops are blocked > 1048.58 sec on osd.19 > 23 ops are blocked > 524.288 sec on osd.19 > 3 osds have slow requests > recovery 221/549658 objects degraded (0.040%) > recovery 221/549658 objects misplaced (0.040%) > pool volumes pg_num 2048 > pgp_num 1400 > > OSD 9 was the one that was the primary when the pg creation process got > stuck. This OSD has been removed and added again (not only osd out but also > removed from the crush map and added again) > > The bad data distribution was probably caused by the low number of pg's > and mainly bad weighing of the OSD. I changed the crush map to give the > same weight to each of the OSD's but that does not change these problems > either: > > ceph osd tree: > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -1 6.50000 pool default > -6 2.00000 host droplet4 > 16 0.25000 osd.16 up 1.00000 1.00000 > 20 0.25000 osd.20 up 1.00000 1.00000 > 21 0.25000 osd.21 up 1.00000 1.00000 > 22 0.25000 osd.22 up 1.00000 1.00000 > 6 0.25000 osd.6 up 1.00000 1.00000 > 18 0.25000 osd.18 up 1.00000 1.00000 > 19 0.25000 osd.19 up 1.00000 1.00000 > 23 0.25000 osd.23 up 1.00000 1.00000 > -5 1.50000 host droplet3 > 3 0.25000 osd.3 up 1.00000 1.00000 > 13 0.25000 osd.13 up 1.00000 1.00000 > 15 0.25000 osd.15 up 1.00000 1.00000 > 4 0.25000 osd.4 up 1.00000 1.00000 > 25 0.25000 osd.25 up 1.00000 1.00000 > 14 0.25000 osd.14 up 1.00000 1.00000 > -2 1.50000 host droplet1 > 7 0.25000 osd.7 up 1.00000 1.00000 > 1 0.25000 osd.1 up 1.00000 1.00000 > 0 0.25000 osd.0 up 1.00000 1.00000 > 9 0.25000 osd.9 up 1.00000 1.00000 > 12 0.25000 osd.12 up 1.00000 1.00000 > 17 0.25000 osd.17 up 1.00000 1.00000 > -4 1.50000 host droplet2 > 10 0.25000 osd.10 up 1.00000 1.00000 > 8 0.25000 osd.8 up 1.00000 1.00000 > 11 0.25000 osd.11 up 1.00000 1.00000 > 2 0.25000 osd.2 up 1.00000 1.00000 > 24 0.25000 osd.24 up 1.00000 1.00000 > 5 0.25000 osd.5 up 1.00000 1.00000 > > I also restarted all OSD's and monitors several times, but no change. The > pool for which the pg is stuck, has replication level 2. I ran out of > things to try. Anyone else something I can try? > > gr, > Bart > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
