Hi Dominique, Is swap enabled on OSD nodes?
My experience is that when swap is enabled on OSD nodes, OSDs can exceed osd_memory_target in a completely unreasonable manner compared to when swap is disabled. Using swap on OSD nodes has been discouraged in the past [1] but I think it's not clearly documented enough. As to understand why these OSDs use so much RAM, read Mark's message from April 11th on this list. It will help you diagnose ram usage. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/releases/nautilus/#notable-changes ----- Le 25 Avr 25, à 16:36, Dominique Ramaekers dominique.ramaek...@cometal.be a écrit : > Hi, > > Housekeeping... I was cleaning up my snapshots and was flattening clones... > Suddenly I ran out of memory on my nodes! > > 4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node > has about 45 GiB of free (available) memory in normal operation. > > After flattening several images and removing dozens of snapshots, the free > memory probably was already lower than the usual 45GiB. I was flattening an > image of 75GiB and 2 out of 4 nodes ran out of memory. One node even > automatically killed processes randomly to free up memory... After regaining > control over the system, I put that node in maintenance and rebooted this > node. > After reboot the free mem was like 70Gib. Over night all of the nodes were > back > on the usual 45GiB. > > Today I checked the free mem of each node: 45GiB free. So I flattened again an > image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast! > > Is there a way to avoid this behavior of the cluster? > > Greetings, > > Dominique. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
