It honestly just looks like your MDS and MGR daemons are not configured to
start automatically. Try starting them manually and then if that fixes the
things, go through and enable them to start automatically. Assuming you
use systemctl the commands to check and fix this would be something like
these. The first one will show you all of the things started with
ceph.target.
sudo systemctl list-dependencies ceph.target
sudo systemctl enable ceph-mgr@servername
sudo systemctl enable ceph-mds@servername
On Mon, Dec 11, 2017 at 1:08 PM Jan Pekař - Imatic <jan.pe...@imatic.cz>
wrote:
> Hi all,
>
> hope that somebody can help me. I have home ceph installation.
> After power failure (it can happen in datacenter also) my ceph booted in
> non-consistent state.
>
> I was backfilling data on one new disk during power failure. First time
> it booted without some OSDs, but I fixed that. Now I have all my OSD's
> running, but cluster state looks like this after some time. :
>
>
> cluster:
> id: 2d9bf17f-3d50-4a59-8359-abc8328fe801
> health: HEALTH_WARN
> 1 filesystem is degraded
> 1 filesystem has a failed mds daemon
> noout,nodeep-scrub flag(s) set
> no active mgr
> 317162/12520262 objects misplaced (2.533%)
> Reduced data availability: 52 pgs inactive, 29 pgs down, 1
> pg peering, 1 pg stale
> Degraded data redundancy: 2099528/12520262 objects degraded
> (16.769%), 427 pgs unclean, 368 pgs degraded, 368 pgs undersized
> 1/3 mons down, quorum imatic-mce-2,imatic-mce
>
> services:
> mon: 3 daemons, quorum imatic-mce-2,imatic-mce, out of quorum: obyvak
> mgr: no daemons active
> mds: cephfs-0/1/1 up , 1 failed
> osd: 8 osds: 8 up, 8 in; 61 remapped pgs
> flags noout,nodeep-scrub
>
> data:
> pools: 8 pools, 896 pgs
> objects: 4446k objects, 9119 GB
> usage: 9698 GB used, 2290 GB / 11988 GB avail
> pgs: 2.455% pgs unknown
> 3.348% pgs not active
> 2099528/12520262 objects degraded (16.769%)
> 317162/12520262 objects misplaced (2.533%)
> 371 stale+active+clean
> 183 active+undersized+degraded
> 154 stale+active+undersized+degraded
> 85 active+clean
> 22 unknown
> 19 stale+down
> 14 stale+active+undersized+degraded+remapped+backfill_wait
> 13 active+undersized+degraded+remapped+backfill_wait
> 10 down
> 6 active+clean+remapped
> 6 stale+active+clean+remapped
> 5 stale+active+remapped+backfill_wait
> 2 active+remapped+backfill_wait
> 2 stale+active+undersized+degraded+remapped+backfilling
> 1 active+undersized+degraded+remapped
> 1 active+undersized+degraded+remapped+backfilling
> 1 stale+peering
> 1 stale+active+clean+scrubbing
>
> There are all OSD's up and running. Before that I completed
> ceph osd out
> on one of my disk and removed that disk from cluster because I don't
> want to use it anymore. It triggered crush reweight and started to
> rebuild my date. I thinkg that should not put my data in danger even I
> saw that some of my PG's were undersized (why?) - but it is not now the
> think.
>
> When I try to do
> ceph pg dump
> I have no response.
>
> But ceph osd dump show weird number of osd's on temporary PG's like
> number 2147483647 <(214)%20748-3647>. I thing that there is some problem
> in some mon or
> other database and peering process cannot complete.
>
> What can I do next? I believed that cluster so much, so I have some data
> I want back. Thank you very much. for help.
>
> My ceph osd dump looks like this:
>
>
> epoch 29442
> fsid 2d9bf17f-3d50-4a59-8359-abc8328fe801
> created 2014-12-10 23:00:49.140787
> modified 2017-12-11 18:54:01.134091
> flags noout,nodeep-scrub,sortbitwise,recovery_deletes
> crush_version 14
> full_ratio 0.97
> backfillfull_ratio 0.91
> nearfull_ratio 0.9
> require_min_compat_client firefly
> min_compat_client firefly
> require_osd_release luminous
> pool 0 'data' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 27537 flags hashpspool
> crash_replay_interval 45 min_read_recency_for_promote 1
> min_write_recency_for_promote 1 stripe_width 0 application cephfs
> pool 1 'metadata' replicated size 3 min_size 1 crush_rule 1 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 27537 flags hashpspool
> min_read_recency_for_promote 1 min_write_recency_for_promote 1
> stripe_width 0 application cephfs
> pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 28088 flags hashpspool
> min_read_recency_for_promote 1 min_write_recency_for_promote 1
> stripe_width 0 application rbd
> removed_snaps [1~5]
> pool 3 'nonreplicated' replicated size 1 min_size 1 crush_rule 2
> object_hash rjenkins pg_num 192 pgp_num 192 last_change 27537 flags
> hashpspool min_read_recency_for_promote 1 min_write_recency_for_promote
> 1 stripe_width 0 application cephfs
> pool 4 'replicated' replicated size 2 min_size 1 crush_rule 0
> object_hash rjenkins pg_num 192 pgp_num 192 last_change 27537 lfor
> 17097/17097 flags hashpspool min_read_recency_for_promote 1
> min_write_recency_for_promote 1 stripe_width 0 application cephfs
> pool 10 'erasure_3_1' erasure size 4 min_size 3 crush_rule 3 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 27537 lfor 9127/9127 flags
> hashpspool tiers 11 read_tier 11 write_tier 11
> min_write_recency_for_promote 1 stripe_width 4128 application cephfs
> pool 11 'erasure_3_1_hot' replicated size 2 min_size 1 crush_rule 1
> object_hash rjenkins pg_num 128 pgp_num 128 last_change 9910 flags
> hashpspool,incomplete_clones tier_of 10 cache_mode writeback
> target_bytes 5368709120 hit_set bloom{false_positive_probability: 0.05,
> target_size: 0, seed: 0} 0s x0 decay_rate 0 search_last_n 1
> min_write_recency_for_promote 1 stripe_width 0
> pool 12 'test' replicated size 1 min_size 1 crush_rule 4 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 27463 flags hashpspool
> stripe_width 0
> max_osd 8
> osd.0 up in weight 1 up_from 29416 up_thru 29433 down_at 29407
> last_clean_interval [29389,29406) 192.168.11.165:6800/9273
> 192.168.11.165:6801/9273 192.168.11.165:6802/9273
> 192.168.11.165:6803/9273 exists,up 630fe0dc-9ec0-456a-bf15-51d6d3ba462d
> osd.1 up in weight 1 up_from 29422 up_thru 29437 down_at 29407
> last_clean_interval [29390,29406) 192.168.11.165:6816/9336
> 192.168.11.165:6817/9336 192.168.11.165:6818/9336
> 192.168.11.165:6819/9336 exists,up ef583c8d-171f-47c4-8a9e-e9eb913cb272
> osd.2 up in weight 1 up_from 29409 up_thru 29433 down_at 29407
> last_clean_interval [29389,29406) 192.168.11.165:6804/9285
> 192.168.11.165:6805/9285 192.168.11.165:6806/9285
> 192.168.11.165:6807/9285 exists,up 1de26ef5-319d-426e-ad75-65aedbbd0328
> osd.3 up in weight 1 up_from 29430 up_thru 29439 down_at 29410
> last_clean_interval [29391,29406) 192.168.11.165:6824/12146
> 192.168.11.165:6825/12146 192.168.11.165:6826/12146
> 192.168.11.165:6827/12146 exists,up 5b63a084-cb0c-4e6a-89c1-9d2fc70cea02
> osd.4 up in weight 1 up_from 29442 up_thru 0 down_at 29347
> last_clean_interval [29317,29343) 192.168.11.165:6828/15193
> 192.168.11.165:6829/15193 192.168.11.165:6830/15193
> 192.168.11.165:6831/15193 exists,up ee9d758d-f2df-41b6-9320-ce89f54c116b
> osd.5 up in weight 1 up_from 29414 up_thru 29431 down_at 29413
> last_clean_interval [29390,29406) 192.168.11.165:6812/9321
> 192.168.11.165:6813/9321 192.168.11.165:6814/9321
> 192.168.11.165:6815/9321 exists,up d1077d42-2c92-4afd-a11e-02fdd59b393b
> osd.6 up in weight 1 up_from 29413 up_thru 29433 down_at 29407
> last_clean_interval [29390,29406) 192.168.11.165:6820/9345
> 192.168.11.165:6821/9345 192.168.11.165:6822/9345
> 192.168.11.165:6823/9345 exists,up f55da9e5-0c03-43fa-af59-56add845c706
> osd.7 up in weight 1 up_from 29422 up_thru 29433 down_at 29407
> last_clean_interval [29389,29406) 192.168.11.165:6808/9309
> 192.168.11.165:6809/9309 192.168.11.165:6810/9309
> 192.168.11.165:6811/9309 exists,up 1e75647b-a1fc-4672-957f-ce5c2b0f4a43
> pg_temp 0.0 [0,5]
> pg_temp 0.2c [0,6]
> pg_temp 0.31 [7,5]
> pg_temp 0.33 [0,2]
> pg_temp 1.3 [1,6,5]
> pg_temp 1.1d [1,3,2]
> pg_temp 1.27 [6,5,2]
> pg_temp 1.2c [3,1,5]
> pg_temp 1.36 [6,3,2]
> pg_temp 1.37 [6,1,5]
> pg_temp 1.38 [6,3,1]
> pg_temp 1.3f [3,5,1]
> pg_temp 4.0 [3,6]
> pg_temp 4.4 [0,6]
> pg_temp 4.9 [0,7]
> pg_temp 4.13 [6,3]
> pg_temp 4.1a [6,5]
> pg_temp 4.41 [6,2]
> pg_temp 4.4b [0,1]
> pg_temp 4.5c [6,0]
> pg_temp 4.76 [0,6]
> pg_temp 4.87 [0,7]
> pg_temp 4.9d [0,3]
> pg_temp 4.a9 [6,7]
> pg_temp 10.1 [0,7,2,1]
> pg_temp 10.2 [3,0,1,7]
> pg_temp 10.5 [2,6,3,0]
> pg_temp 10.7 [6,7,0,3]
> pg_temp 10.a [0,3,5,6]
> pg_temp 10.c [7,6,5,0]
> pg_temp 10.f [7,3,6,0]
> pg_temp 10.1c [6,1,0,3]
> pg_temp 10.25 [0,7,5,3]
> pg_temp 10.26 [3,2,5,1]
> pg_temp 10.29 [7,5,2,0]
> pg_temp 10.2f [1,5,7,0]
> pg_temp 10.3b [7,3,0,6]
> pg_temp 10.41 [0,1,3,7]
> pg_temp 10.47 [3,0,7,5]
> pg_temp 10.4c [7,3,5,1]
> pg_temp 10.51 [1,7,0,2]
> pg_temp 10.54 [3,0,7,5]
> pg_temp 10.55 [7,1,0,3]
> pg_temp 10.5a [7,0,3,2]
> pg_temp 10.5b [2147483647 <(214)%20748-3647>,0,5,6]
> pg_temp 10.5e [7,2,3,0]
> pg_temp 10.5f [6,3,2,1]
> pg_temp 10.63 [7,5,3,2]
> pg_temp 10.64 [3,6,7,1]
> pg_temp 10.66 [0,1,5,7]
> pg_temp 10.6c [7,3,2,0]
> pg_temp 10.6f [6,3,2,7]
> pg_temp 10.70 [0,7,6,1]
> pg_temp 10.72 [7,6,1,3]
> pg_temp 10.73 [6,2147483647 <(214)%20748-3647>,7,0]
> pg_temp 10.74 [7,1,6,3]
> pg_temp 10.7c [0,6,1,7]
> pg_temp 10.7f [0,7,1,2]
> pg_temp 11.27 [6,5]
> pg_temp 11.38 [3,6]
> pg_temp 11.78 [3,1]
>
> Thank you all for help. It is important for me.
> Jan Pekar
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
