Dear all, we finally managed to collect perf data and it seems to show a smoking gun. Since this thread is already heavily cluttered on lists.ceph.io I started a new one: "MDS hung in purge_stale_snap_data after populating cache" (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XLYHTZBD4AXNPFZOLG7NG24Z4DWXIARG/).
Please take a look at this post. Thanks and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frédéric Nass <frederic.n...@univ-lorraine.fr> Sent: Tuesday, January 14, 2025 8:25 AM To: Frank Schilder; ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Subject: Re: [ceph-users] Re: Help needed, ceph fs down due to large stray dir Hi Frank, More than ever. You should open a tracker and post debug logs there so anyone can have a look. Regards, Frédéric. ________________________________ De : Frank Schilder <fr...@dtu.dk> Envoyé : lundi 13 janvier 2025 17:39 À : ceph-users@ceph.io Cc: Dan van der Ster; Patrick Donnelly; Bailey Allison; Spencer Macphee Objet : [ceph-users] Re: Help needed, ceph fs down due to large stray dir Dear all, a quick update and some answers. We set up a dedicated host for running an MDS and debugging the problem. On this host we have 750G RAM, 4T swap and 4T log, both on fast SSDs. Plan is to monitor with "perf top" the MDS becoming the designated MDS for the problematic rank and also pull out a detailed log about the startup until the MDS hangs. I have some questions about that, a new observation that might be relevant and some answers to some suggestions, in that order. I need to install debug info for perf to give useful output. I can't find a meta package that has all ceph-debuginfo rpms as a dependency. Which ones should I install? Also, should I install some kernel debug info packages? Please note that every restart takes about 1h to hit the issue. I would like to have as much as possible installed the first time. A new observation: After every restart cycle the rank loads a little bit less into cache. However, the num_stray count does not decrease. Could that mean the problem is not the high num_stray count but something else? Answer to a suggestion: It is not possible to access anything on the MDS or the entire file system. Hence, trying to stat some files/dirs is not possible. Furthermore, not all stray items can be reintegrated with this method and I'm afraid our stray items are mostly of this nature. This means that in octopus the only way to trim (evaluate) stray items was an MDS restart. For details, the relevant discussions are https://www.spinics.net/lists/ceph-users/msg70459.html and https://www.spinics.net/lists/ceph-users/msg73150.html with the most important info in this message: https://www.spinics.net/lists/ceph-users/msg70849.html . Summary: As part of the debugging back then I executed a recursive stat-ing of files and directories on the *entire* file system only to observe that the stray count didn't change. This was when Gregory finally explained that hard links can block stray removal on snaptrim also for paths that are no longer accessible through the file system or any snapshots, that is, the usual stray evaluation doesn't have any effect. That's the situation we are in, we need the MDS do it itself. A correction: It was actually Venky Shankar participating in this communication and not Neha. Is Venkhy still working on the FS? Thanks for package hints and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
