Hi Frank, We had this long ago related to a user generating lots of hard links. Snapshots will have a similar effect. (in these cases, if a user deletes the original file, the file goes into stray until it is "reintegrated").
If you can find the dir where they're working, `ls -lR` will force those to reintegrate (you will see because the num strays will drop back down). You might have to ls -lR in a snap directory, or in the current tree -- you have to browse around and experiment. pacific does this re-integration automatically. -- dan On Fri, Jan 14, 2022 at 12:24 PM Frank Schilder <fr...@dtu.dk> wrote: > > Hi Venky, > > thanks for your reply. I think the first type of messages was a race > condition. A user was running rm and find on the same folder at the same > time. The second type of message (duplicate inode in stray) might point to an > a bit more severe issue. For a while now I observe that > ".mds_cache.num_strays" is really large and, on average, constantly > increasing: > > # ssh ceph-08 'ceph daemon mds.$(hostname -s) perf dump | jq > .mds_cache.num_strays' > 1081531 > > This is by no means justified by people deleting files. Our snapshots rotate > completely every 3 days and the stray buckets should get purged regularly. I > have 2 questions: > > 1) Would a "cephfs-data-scan scan_links" detect and potentially resolve this > problem (orphaned inodes in stray bucket)? > 2) For a file system of our size, how long would a "cephfs-data-scan > scan_links" run approximately (I need to estimate downtime)? I think I can > execute up to 35-40 workers. The fs size is: > > ceph.dir.rbytes="2078289930815425" > ceph.dir.rentries="278320382" > > Thanks for your help! > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Venky Shankar <vshan...@redhat.com> > Sent: 12 January 2022 12:24 > To: Frank Schilder > Cc: ceph-users > Subject: Re: [ceph-users] cephfs: [ERR] loaded dup inode > > On Tue, Jan 11, 2022 at 6:07 PM Frank Schilder <fr...@dtu.dk> wrote: > > > > Hi all, > > > > I found a bunch of error messages like below in our ceph log (2 different > > types). How bad is this and should I do something? > > > > Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic > > (stable). > > > > 2022-01-11 11:49:47.687010 [ERR] loaded dup inode 0x10011bac31c [4f8,head] > > v1046724308 at ~mds0/stray1/10011bac31c, but inode 0x10011bac31c.head > > v1046760378 already exists at > > [...]/miniconda3/envs/ffpy_gwa3/lib/python3.6/site-packages/python_dateutil-2.8.0.dist-info/INSTALLER > > > > 2022-01-11 11:49:47.682346 [ERR] loaded dup inode 0x10011bac7fc [4f8,head] > > v1046725418 at ~mds0/stray1/10011bac7fc, but inode 0x10011bac7fc.head > > v1046760674 already exists at ~mds0/stray2/10011bac7fc > > I've seen this earlier. Not sure how we end up with an inode in two > stray directories, but it doesn't look serious. > > You could try stopping all MDSs and run `cephfs-data-scan scan_links` > (courtesy Zheng) to see if the errors go away. > > > > > Best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > -- > Cheers, > Venky > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
