Hey Kenneth,
We encountered this when the number of strays (unlinked files yet to be
purged) reached 1 million, which is a result of many many file removals
happening on the fs repeatedly. It can also happen when there are more than
100k files in a dir with default settings.
You can tune it via 'mds_bal_fragment_size_max' setting on the mds either
temporarily to rm files or permanently. Beware setting it too high.
Check num strays in mds cache by running `ceph daemon mds.{mds name} perf
dump` and inspecting the mds cache section for num_strays. The 1 million
limit is a multiple/function of the mds bal fragment size (10x).
Raf
On Fri, May 10, 2019, 9:03 PM Kenneth Waegeman <kenneth.waege...@ugent.be>
wrote:
> Hi all,
>
> I am seeing issues on cephfs running 13.2.5 when deleting files:
>
> [root@osd006 ~]# rm /mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700
> rm: remove regular empty file
> ‘/mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700’? y
> rm: cannot remove
> ‘/mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700’: No space left
> on device
>
> few minutes later, I can remove it without problem. This happens
> especially when there are a lot of files deleted somewhere on the
> filesystem around the same time.
>
> We already have tuned our mds config:
>
> [mds]
> mds_cache_memory_limit=10737418240
> mds_log_max_expiring=200
> mds_log_max_segments=200
> mds_max_purge_files=2560
> mds_max_purge_ops=327600
> mds_max_purge_ops_per_pg=20
>
> ceph -s is reporting everything clean, and the file system space usage
> is less than 50%, also no full osds or anything.
>
> Is there a way to further debug what the bottleneck is when removing
> files that gives this 'no space left on device' error?
>
>
> Thank you very much!
>
> Kenneth
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
