With ceph health detail
get a list of problematic pgs with ceph pg <pg_id> query check why the pgs are stuck. Check the log files of all OSDs on that restart where the restart triggered the problem. Saverio 2017-06-02 14:16 GMT+02:00 Grant Morley <grantmorley1...@gmail.com>: > We are using Ceph Jewel (10.2.7) running on Ubuntu 14.04LTS > > osd_recovery_max_active": "1" > osd_max_backfills": "1" > osd_recovery_op_priority": "3" > > Limit Soft Limit Hard Limit Units > Max cpu time unlimited unlimited seconds > Max file size unlimited unlimited bytes > Max data size unlimited unlimited bytes > Max stack size 8388608 unlimited bytes > Max core file size 0 unlimited bytes > Max resident set unlimited unlimited bytes > Max processes 256369 256369 > processes > Max open files 327680 327680 files > Max locked memory 65536 65536 bytes > Max address space unlimited unlimited bytes > Max file locks unlimited unlimited locks > Max pending signals 256369 256369 signals > Max msgqueue size 819200 819200 bytes > Max nice priority 0 0 > Max realtime priority 0 0 > Max realtime timeout unlimited unlimited us > > We did try changing the osd_recovery_max_active to "3" but that seemed tlo > make things run slower > > Thanks, > > On Fri, Jun 2, 2017 at 1:08 PM, Saverio Proto <ziopr...@gmail.com> wrote: >> >> To give you some help you need to tell us the ceph version you are >> using and from ceph.conf in the section [osd] what values you have for >> the following ? >> >> [osd] >> osd max backfills >> osd recovery max active >> osd recovery op priority >> >> these three settings can influence the recovery speed. >> >> Also, do you have big enough limits ? >> >> Check on any host the content of: /proc/`pid_of_the_osd`/limits >> >> >> Saverio >> >> 2017-06-02 14:00 GMT+02:00 Grant Morley <grantmorley1...@gmail.com>: >> > HEALTH_ERR 210 pgs are stuck inactive for more than 300 seconds; 296 pgs >> > backfill_wait; 3 pgs backfilling; 1 pgs degraded; 202 pgs peering; 1 pgs >> > recovery_wait; 1 pgs stuck degraded; 210 pgs stuck inactive; 510 pgs >> > stuck >> > unclean; 3308 requests are blocked > 32 sec; 41 osds have slow requests; >> > recovery 2/11091408 objects degraded (0.000%); recovery 1778127/11091408 >> > objects misplaced (16.032%); nodown,noout,noscrub,nodeep-scrub flag(s) >> > set >> > >> > pg 3.235 is stuck inactive for 138232.508429, current state peering, >> > last >> > acting [11,26,1] >> > pg 1.237 is stuck inactive for 138260.482588, current state peering, >> > last >> > acting [8,41,34] >> > pg 2.231 is stuck inactive for 138258.316031, current state peering, >> > last >> > acting [24,53,8] >> > pg 2.22e is stuck inactive for 194033.321591, current state >> > remapped+peering, last acting [0,29,1] >> > pg 1.22c is stuck inactive for 102514.200154, current state peering, >> > last >> > acting [51,7,20] >> > pg 2.228 is stuck inactive for 138258.317797, current state peering, >> > last >> > acting [53,4,34] >> > pg 1.227 is stuck inactive for 138258.244681, current state >> > remapped+peering, last acting [48,35,11] >> > pg 2.220 is stuck inactive for 193940.066322, current state >> > remapped+peering, last acting [9,39,8] >> > pg 1.222 is stuck inactive for 101474.087688, current state peering, >> > last >> > acting [23,11,35] >> > pg 3.130 is stuck inactive for 99735.451290, current state peering, last >> > acting [27,37,17] >> > pg 3.136 is stuck inactive for 138221.552865, current state peering, >> > last >> > acting [26,49,10] >> > pg 3.13c is stuck inactive for 137563.906503, current state peering, >> > last >> > acting [51,53,7] >> > pg 2.142 is stuck inactive for 99962.462932, current state peering, last >> > acting [37,16,34] >> > pg 1.141 is stuck inactive for 138257.572476, current state >> > remapped+peering, last acting [5,17,49] >> > pg 2.141 is stuck inactive for 102567.745720, current state peering, >> > last >> > acting [36,7,15] >> > pg 3.144 is stuck inactive for 138218.289585, current state >> > remapped+peering, last acting [18,28,16] >> > pg 1.14d is stuck inactive for 138260.030530, current state peering, >> > last >> > acting [46,43,17] >> > pg 3.155 is stuck inactive for 138227.368541, current state >> > remapped+peering, last acting [33,20,52] >> > pg 2.8d is stuck inactive for 100251.802576, current state peering, last >> > acting [6,39,27] >> > pg 2.15c is stuck inactive for 102567.512279, current state >> > remapped+peering, last acting [7,35,49] >> > pg 2.167 is stuck inactive for 138260.093367, current state peering, >> > last >> > acting [35,23,17] >> > pg 3.9d is stuck inactive for 117050.294600, current state peering, last >> > acting [12,51,23] >> > pg 2.16e is stuck inactive for 99846.214239, current state peering, last >> > acting [25,5,8] >> > pg 2.17b is stuck inactive for 99733.504794, current state peering, last >> > acting [49,27,14] >> > pg 3.178 is stuck inactive for 99973.600671, current state peering, last >> > acting [29,16,40] >> > pg 3.240 is stuck inactive for 28768.488851, current state >> > remapped+peering, >> > last acting [33,8,32] >> > pg 3.b6 is stuck inactive for 138222.461160, current state peering, last >> > acting [26,29,34] >> > pg 2.17e is stuck inactive for 159229.154401, current state peering, >> > last >> > acting [13,42,48] >> > pg 2.17c is stuck inactive for 104921.767401, current state >> > remapped+peering, last acting [23,12,24] >> > pg 3.17d is stuck inactive for 137563.979966, current state >> > remapped+peering, last acting [43,24,29] >> > pg 1.24b is stuck inactive for 93144.933177, current state peering, last >> > acting [43,20,37] >> > pg 1.bd is stuck inactive for 102616.793475, current state peering, last >> > acting [16,30,35] >> > pg 3.1d6 is stuck inactive for 99974.485247, current state peering, last >> > acting [16,38,29] >> > pg 2.172 is stuck inactive for 193919.627310, current state inactive, >> > last >> > acting [39,21,10] >> > pg 1.171 is stuck inactive for 104947.558748, current state peering, >> > last >> > acting [49,9,25] >> > pg 1.243 is stuck inactive for 208452.393430, current state peering, >> > last >> > acting [45,32,24] >> > pg 3.aa is stuck inactive for 104958.230601, current state >> > remapped+peering, >> > last acting [51,12,13] >> > >> > 41 osds have slow requests >> > recovery 2/11091408 objects degraded (0.000%) >> > recovery 1778127/11091408 objects misplaced (16.032%) >> > nodown,noout,noscrub,nodeep-scrub flag(s) set >> > >> > That is what we seem to be getting a lot of. It appears the PG's are >> > just >> > stuck as inactive. I am not sure how to get around that. >> > >> > Thanks, >> > >> > On Fri, Jun 2, 2017 at 12:55 PM, Saverio Proto <ziopr...@gmail.com> >> > wrote: >> >> >> >> Usually 'ceph health detail' gives better info on what is making >> >> everything stuck. >> >> >> >> Saverio >> >> >> >> 2017-06-02 13:51 GMT+02:00 Grant Morley <grantmorley1...@gmail.com>: >> >> > Hi All, >> >> > >> >> > I wonder if anyone could help at all. >> >> > >> >> > We were doing some routine maintenance on our ceph cluster and after >> >> > running >> >> > a "service ceph-all restart" on one of our nodes we noticed that >> >> > something >> >> > wasn't quite right. The cluster has gone into an error mode and we >> >> > have >> >> > multiple stuck PGs and the object replacement recovery is taking a >> >> > strangely >> >> > long time. At first there was about 46% objects misplaced and we now >> >> > have >> >> > roughly 16%. >> >> > >> >> > However it has taken about 36 hours to do the recovery so far and >> >> > with a >> >> > possible 16 to go we are looking at a fairly major issue. As a lot of >> >> > the >> >> > system is now blocked for read / writes, customers cannot access >> >> > their >> >> > VMs. >> >> > >> >> > I think the main issue at the moment is that we have 210pgs stuck >> >> > inactive >> >> > and nothing we seem to do can get them to peer. >> >> > >> >> > Below is an ouptut of the ceph status. Can anyone help or have any >> >> > ideas >> >> > on >> >> > how to speed up the recover process? We have tried turning down >> >> > logging >> >> > on >> >> > the OSD's but some are going so slow they wont allow us to injectargs >> >> > into >> >> > them. >> >> > >> >> > health HEALTH_ERR >> >> > 210 pgs are stuck inactive for more than 300 seconds >> >> > 298 pgs backfill_wait >> >> > 3 pgs backfilling >> >> > 1 pgs degraded >> >> > 200 pgs peering >> >> > 1 pgs recovery_wait >> >> > 1 pgs stuck degraded >> >> > 210 pgs stuck inactive >> >> > 512 pgs stuck unclean >> >> > 3310 requests are blocked > 32 sec >> >> > recovery 2/11094405 objects degraded (0.000%) >> >> > recovery 1785063/11094405 objects misplaced (16.090%) >> >> > nodown,noout,noscrub,nodeep-scrub flag(s) set >> >> > >> >> > election epoch 16314, quorum 0,1,2,3,4,5,6,7,8 >> >> > >> >> > >> >> > storage-1,storage-2,storage-3,storage-4,storage-5,storage-6,storage-7,storage-8,storage-9 >> >> > osdmap e213164: 54 osds: 54 up, 54 in; 329 remapped pgs >> >> > flags nodown,noout,noscrub,nodeep-scrub >> >> > pgmap v41030942: 2036 pgs, 14 pools, 14183 GB data, 3309 >> >> > kobjects >> >> > 43356 GB used, 47141 GB / 90498 GB avail >> >> > 2/11094405 objects degraded (0.000%) >> >> > 1785063/11094405 objects misplaced (16.090%) >> >> > 1524 active+clean >> >> > 298 active+remapped+wait_backfill >> >> > 153 peering >> >> > 47 remapped+peering >> >> > 10 inactive >> >> > 3 active+remapped+backfilling >> >> > 1 active+recovery_wait+degraded+remapped >> >> > >> >> > Many thanks, >> >> > >> >> > Grant >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > OpenStack-operators mailing list >> >> > OpenStack-operators@lists.openstack.org >> >> > >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> > >> > >> > > > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
