Ok, lesson learned the hard way. Thank goodness it was a test cluster. Thanks a lot Bryan!
El jue., 17 ene. 2019 a las 21:46, Bryan Stillwell (<bstillw...@godaddy.com>) escribió: > When you use 3+2 EC that means you have 3 data chunks and 2 erasure chunks > for your data. So you can handle two failures, but not three. The > min_size setting is preventing you from going below 3 because that's the > number of data chunks you specified for the pool. I'm sorry to say this, > but since the data was wiped off the other 3 nodes there isn't anything > that can be done to recover it. > > > > Bryan > > > > > > *From: *ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of Félix > Barbeira <fbarbe...@gmail.com> > *Date: *Thursday, January 17, 2019 at 1:27 PM > *To: *Ceph Users <ceph-users@lists.ceph.com> > *Subject: *[ceph-users] How to reduce min_size of an EC pool? > > > > I want to bring back my cluster to HEALTHY state because right now I have > not access to the data. > > > > I have an 3+2 EC pool on a 5 node cluster. 3 nodes were lost, all data > wiped. They were reinstalled and added to cluster again. > > > > The "ceph health detail" command says to reduce min_size number to a value > lower than 3, but: > > > > root@ceph-monitor02:~# ceph osd pool set default.rgw.buckets.data > min_size 2 > > Error EINVAL: pool min_size must be between 3 and 5 > > root@ceph-monitor02:~# > > > > This is the situation: > > > > root@ceph-monitor01:~# ceph -s > > cluster: > > id: ce78b02d-03df-4f9e-a35a-31b5f05c4c63 > > health: HEALTH_WARN > > Reduced data availability: 515 pgs inactive, 512 pgs incomplete > > > > services: > > mon: 3 daemons, quorum ceph-monitor01,ceph-monitor03,ceph-monitor02 > > mgr: ceph-monitor02(active), standbys: ceph-monitor01, ceph-monitor03 > > osd: 57 osds: 57 up, 57 in > > > > data: > > pools: 8 pools, 568 pgs > > objects: 4.48 M objects, 10 TiB > > usage: 24 TiB used, 395 TiB / 419 TiB avail > > pgs: 0.528% pgs unknown > > 90.141% pgs not active > > 512 incomplete > > 53 active+clean > > 3 unknown > > > > root@ceph-monitor01:~# > > > > And this is the output of health detail: > > > > root@ceph-monitor01:~# ceph health detail > > HEALTH_WARN Reduced data availability: 515 pgs inactive, 512 pgs incomplete > > PG_AVAILABILITY Reduced data availability: 515 pgs inactive, 512 pgs > incomplete > > pg 10.1cd is stuck inactive since forever, current state incomplete, > last acting [9,48,41,58,17] (reducing pool default.rgw.buckets.data > min_size from 3 may help; search ceph.com/docs for 'incomplete') > > pg 10.1ce is incomplete, acting [3,13,14,42,21] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1cf is incomplete, acting [36,27,3,39,51] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d0 is incomplete, acting [29,9,38,4,56] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d1 is incomplete, acting [2,34,17,7,30] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d2 is incomplete, acting [41,45,53,13,32] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d3 is incomplete, acting [7,28,15,20,3] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d4 is incomplete, acting [11,40,25,23,0] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d5 is incomplete, acting [32,51,20,57,28] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d6 is incomplete, acting [2,53,8,16,15] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d7 is incomplete, acting [1,2,33,43,42] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d8 is incomplete, acting [27,49,9,48,20] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1d9 is incomplete, acting [37,8,7,11,20] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1da is incomplete, acting [27,14,33,15,53] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1db is incomplete, acting [58,53,6,26,4] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1dc is incomplete, acting [21,12,47,35,19] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1dd is incomplete, acting [51,4,52,24,7] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1de is incomplete, acting [38,29,21,41,44] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1df is incomplete, acting [25,4,30,61,11] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e0 is incomplete, acting [27,57,21,6,13] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e1 is incomplete, acting [8,7,25,15,29] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e2 is incomplete, acting [49,37,62,11,31] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e3 is incomplete, acting [1,49,32,56,48] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e4 is incomplete, acting [16,23,37,18,20] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e5 is incomplete, acting [21,38,6,23,57] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e6 is incomplete, acting [44,32,11,15,41] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e7 is incomplete, acting [35,20,42,48,26] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e8 is incomplete, acting [49,41,16,19,5] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1e9 is incomplete, acting [26,17,58,20,24] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ea is incomplete, acting [57,23,25,26,12] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1eb is incomplete, acting [39,30,61,18,10] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ec is incomplete, acting [21,20,11,38,4] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ed is incomplete, acting [56,34,45,42,33] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ee is incomplete, acting [40,53,2,27,33] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ef is incomplete, acting [21,56,3,39,42] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f0 is incomplete, acting [32,49,45,19,2] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f1 is incomplete, acting [46,34,45,8,47] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f2 is incomplete, acting [43,39,20,30,16] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f3 is incomplete, acting [30,43,23,25,32] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f4 is incomplete, acting [30,16,29,2,8] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f5 is incomplete, acting [15,28,6,11,7] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f6 is incomplete, acting [61,25,45,34,33] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f7 is incomplete, acting [33,27,6,11,15] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f8 is incomplete, acting [47,8,30,19,7] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1f9 is incomplete, acting [11,44,58,26,20] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1fa is incomplete, acting [32,51,19,39,2] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1fb is incomplete, acting [14,19,61,35,30] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1fc is incomplete, acting [37,0,47,17,18] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1fd is incomplete, acting [49,20,34,62,15] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1fe is incomplete, acting [46,52,33,34,9] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > pg 10.1ff is incomplete, acting [33,21,7,19,52] (reducing pool > default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs > for 'incomplete') > > root@ceph-monitor02:~# > > > > Somebody has an idea of how to fix this?? > > > > Maybe copying the data to a replicated pool with min_size=1 ? > > > > All data are hopelessly lost? > > > > Thanks in advance. > > -- > > Félix Barbeira. > -- Félix Barbeira.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
