[ceph-users] Re: Full OSD with almost empty pools

[Adam Prycki]

Hi Igor,

It looks like you're right.

ceph tell osd.1750 status
{
    "cluster_fsid": "bec60cda-a306-11ed-abd9-75488d4e8f4a",
    "osd_fsid": "388906f8-1df8-45a2-9895-067ee2e0c055",
    "whoami": 1750,
    "state": "active",
    "maps": "[1502335~261370]",
    "oldest_map": "1502335",
    "newest_map": "1763704",
    "cluster_osdmap_trim_lower_bound": 1502335,
    "num_pgs": 0
}

One of the OSDs I've checked has about 261K maps
Could this cause bluestore to grow to ~380GiB?
What settings could affect number of mapps stored on OSD.




Evacuating all PGs from OSD doesn't lower it's usage.
For example

I assume it's bluestore and not PG/Crush related issue.

Best regards
Adam

W dniu 5.05.2025 o 16:20, Igor Fedotov pisze:

Only think that comes to mind is mon_min_osdmap_epochs which I configured to 2000 a while ago. We are also running long rebalancing on this cluster. At the beginning of the year we've expended this cluster and we are still waiting for HDD pool to finish rebalancing. We have only 2% of data left to rebalance. It's taking so long because I wanted to stay with mclock scheduler at first. Rebalancing shouldn't affect SSD pool directly, but it keeps hdd pools from being 100% active+clean 1814 ssd 0 1.00000 447 GiB 382 GiB 381 GiB 351 MiB 1.0 GiB 65 GiB 85.44 1.00 0 up

Hi Adam,





Having a delta in hundreds of thousands of epochs would confirm the case.


Thanks,

Igor

On 05.05.2025 16:41, Adam Prycki wrote:

I recall a case where pretty high OSD utilization was caused by no trimming for osdmaps. You might want to check the output of 'ceph tell osd.N status' command and compare oldest_map vs. newest_map values.

Sorry guys, I've forgotten to add attachments to my previous email.

In attachments:
ceph osd crush rule dump
ceph osd df tree

Best regards
Adam Prycki


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

smime.p7s
Description: Kryptograficzna sygnatura S/MIME

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io