Im not sure that setup to 128 max backfills is a good idea, I shared our config
for recovery and backfilling:
osd recovery threads = 4
osd recovery op priority = 1
osd recovery max active = 2
osd recovery max single start = 1
osd max backfills = 4
osd backfill scan max = 16
osd backfill scan min = 4
osd client op priority = 63
Check the fullest osd my be one is full and this prevent of use the other 131TB
raw or distribution is not even across osd
Check mon osd full ratio and mon osd nearfull ratio , maybe put just a 2% more
in full ratio makes different for you.
Regards
De: ceph-users <ceph-users-boun...@lists.ceph.com> En nombre de Kári Bertilsson
Enviado el: jueves, 9 de mayo de 2019 14:08
Para: ceph-users <ceph-users@lists.ceph.com>
Asunto: [ceph-users] Getting "No space left on device" when reading from cephfs
Hello
I am running cephfs with 8/2 erasure coding. I had about 40tb usable free(110tb
raw), one small disk crashed and i added 2x10tb disks. Now it's backfilling &
recovering with 0B free and i can't read a single file from the file system...
This happend with max-backfilling 4, but i have increased max backfills to 128,
to hopefully get this over a little faster since system has been unusable for
12 hours anyway. Not sure yet if that was a good idea.
131TB of raw space was somehow not enough to keep things running. Any tips to
avoid this kind of scenario in the future ?
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
489TiB 131TiB 358TiB 73.17
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
ec82_pool 41 278TiB 100.00 0B 28549450
cephfs_metadata 42 174MiB 0.04 381GiB 666939
rbd 51 99.3GiB 20.68 381GiB 25530
data:
pools: 3 pools, 704 pgs
objects: 29.24M objects, 278TiB
usage: 358TiB used, 131TiB / 489TiB avail
pgs: 1265432/287571907 objects degraded (0.440%)
12366014/287571907 objects misplaced (4.300%)
536 active+clean
137 active+remapped+backfilling
27 active+undersized+degraded+remapped+backfilling
4 active+remapped+backfill_toofull
io:
client: 64.0KiB/s wr, 0op/s rd, 7op/s wr
recovery: 1.17GiB/s, 113objects/s
Is there anything i can do to restore reading ? I can understand writing not
working, but why is it blocking reading also ? Any tips ?
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
