Looks like it's overloaded and runs into a timeout. For a test/dev
environment: try to set the nodown flag for this experiment if you just
want to ignore these timeouts completely.


Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


On Wed, Jun 26, 2019 at 1:26 PM zhanrzh...@teamsun.com.cn <
zhanrzh...@teamsun.com.cn> wrote:

> Hi,all:
>     I start ceph cluster on my machine with development mode,to estimate
> the time of recoverying after increasing pgp_num.
>    all of daemon  run on one machine.
>     CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
>     memory: 377GB
>     OS:CentOS Linux release 7.6.1810
>     ceph version:hammer
>
> builded ceph according to
> http://docs.ceph.com/docs/hammer/dev/quick_guide/,
> ceph -s shows:
>     cluster 15ec2f3f-86e5-46bc-bf98-4b35841ee6a5
>      health HEALTH_WARN
>             pool rbd pg_num 512 > pgp_num 256
>      monmap e1: 1 mons at {a=172.30.250.25:6789/0}
>             election epoch 2, quorum 0 a
>      osdmap e88: 30 osds: 30 up, 30 in
>       pgmap v829: 512 pgs, 1 pools, 57812 MB data, 14454 objects
>             5691 GB used, 27791 GB / 33483 GB avail
>                  512 active+clean
> and ceph osd tree[3]
> It start to recovering after i increased pgp_num. ceph -w says there are
> some osd down, but the process is runing.All configuration items of osd or
> mon are default[1]
> some messages that ceph -w[2] says,as below :
>
> 2019-06-26 15:03:21.839750 mon.0 [INF] pgmap v842: 512 pgs: 127
> active+degraded, 84 activating+degraded, 256 active+clean, 45
> active+recovering+degraded; 57812 MB data, 5714 GB used, 27769 GB / 33483
> GB avail; 22200/43362 objects degraded (51.197%); 50789 kB/s, 12 objects/s
> recovering
> 2019-06-26 15:03:21.840884 mon.0 [INF] osd.1 172.30.250.25:6804/22500
> failed (3 reports from 3 peers after 24.867116 >= grace 20.000000)
> 2019-06-26 15:03:21.841459 mon.0 [INF] osd.9 172.30.250.25:6836/25078
> failed (3 reports from 3 peers after 24.867645 >= grace 20.000000)
> 2019-06-26 15:03:21.841709 mon.0 [INF] osd.0 172.30.250.25:6800/22260
> failed (3 reports from 3 peers after 24.846423 >= grace 20.000000)
> 2019-06-26 15:03:21.842286 mon.0 [INF] osd.13 172.30.250.25:6852/26651
> failed (3 reports from 3 peers after 24.846896 >= grace 20.000000)
> 2019-06-26 15:03:21.842607 mon.0 [INF] osd.5 172.30.250.25:6820/23661
> failed (3 reports from 3 peers after 24.804869 >= grace 20.000000)
> 2019-06-26 15:03:21.842938 mon.0 [INF] osd.10 172.30.250.25:6840/25490
> failed (3 reports from 3 peers after 24.805155 >= grace 20.000000)
> 2019-06-26 15:03:21.843134 mon.0 [INF] osd.12 172.30.250.25:6848/26277
> failed (3 reports from 3 peers after 24.805329 >= grace 20.000000)
> 2019-06-26 15:03:21.843591 mon.0 [INF] osd.8 172.30.250.25:6832/24722
> failed (3 reports from 3 peers after 24.805843 >= grace 20.000000)
> 2019-06-26 15:03:21.849664 mon.0 [INF] osd.21 172.30.250.25:6884/29762
> failed (3 reports from 3 peers after 23.497080 >= grace 20.000000)
> 2019-06-26 15:03:21.862729 mon.0 [INF] osd.14 172.30.250.25:6856/27025
> failed (3 reports from 3 peers after 23.510172 >= grace 20.000000)
> 2019-06-26 15:03:21.864222 mon.0 [INF] osdmap e91: 30 osds: 29 up, 30 in
> 2019-06-26 15:03:20.336758 osd.11 [WRN] map e91 wrongly marked me down
> 2019-06-26 15:03:23.408659 mon.0 [INF] pgmap v843: 512 pgs: 8
> stale+activating+degraded, 8 stale+active+clean, 161 active+degraded, 2
> stale+active+recovering+degraded, 33 activating+degraded, 248 active+clean,
> 45 active+recovering+degraded, 7 stale+active+degraded; 57812 MB data, 5730
> GB used, 27752 GB / 33483 GB avail; 27317/43362 objects degraded (62.998%);
> 61309 kB/s, 14 objects/s recovering
> 2019-06-26 15:03:27.538229 mon.0 [INF] osd.18 172.30.250.25:6872/28632
> failed (3 reports from 3 peers after 23.180489 >= grace 20.000000)
> 2019-06-26 15:03:27.539416 mon.0 [INF] osd.7 172.30.250.25:6828/24366
> failed (3 reports from 3 peers after 21.900054 >= grace 20.000000)
> 2019-06-26 15:03:27.541831 mon.0 [INF] osdmap e92: 30 osds: 19 up, 30 in
> 2019-06-26 15:03:32.748179 mon.0 [INF] osdmap e93: 30 osds: 17 up, 30 in
> 2019-06-26 15:03:33.678682 mon.0 [INF] pgmap v845: 512 pgs: 17
> stale+activating+degraded, 95 stale+active+clean, 55 active+degraded, 13
> peering, 18 stale+active+recovering+degraded, 20 activating+degraded, 155
> active+clean, 22 active+recovery_wait+degraded, 48
> active+recovering+degraded, 69 stale+active+degraded; 57812 MB data, 5734
> GB used, 27748 GB / 33483 GB avail; 26979/43362 objects degraded (62.218%);
> 11510 kB/s, 2 objects/s recovering
> 2019-06-26 15:03:33.775701 osd.1 [WRN] map e92 wrongly marked me down
>
> Has anyone got any thoughts on what might have happened, or tips on how to
> dig further into this?
>
> [1] https://github.com/rongzhen-zhan/myfile/blob/master/osd.0.conf
> [2] https://github.com/rongzhen-zhan/myfile/blob/master/ceph-watch.txt
> [3] https://github.com/rongzhen-zhan/myfile/blob/master/ceph%20osd%20tree
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to