I have tested with the following kernels and see this issue in each case. linux-image-5.10.0-16-amd64 5.10.127-1 amd64 Linux 5.10 for 64-bit PCs (signed) linux-image-5.15.0-0.bpo.3-amd64 5.15.15-2~bpo11+1 amd64 Linux 5.15 for 64-bit PCs (signed) linux-image-5.18.0-0.deb11.3-amd64 5.18.14-1~bpo11+1 amd64 Linux 5.18 for 64-bit PCs (signed)
An interesting note is that when using the 5.18 kernel, I had to run echo 3 > /proc/sys/vm/drop_caches to resolve the issue. echo 2 > /proc/sys/vm/drop_caches did not work as it did on the 5.10 and 5.15 kernels. > -----Original Message----- > From: Jason Breitman > Sent: Friday, August 26, 2022 3:36 PM > To: 'Ben Hutchings' <b...@decadent.org.uk>; '1017...@bugs.debian.org' > <1017...@bugs.debian.org> > Subject: RE: Bug#1017720: nfs-common: No such file or directory > > I was able to identify another workaround today which may help you to > identify the issue. > The workaround is to touch the directory where the troubled files live on the > file server. > I believe this tells us that updating the modify time attribute is used by the > cache. > It should be noted that access time updates are disabled on the file server. > > I also wanted to restate that we use rsync to push out these application > updates and also use rsync to sync data files. > Our rsync options preserve timestamps, so it is possible that the new files > have an older timestamp than "now". > It is not the case that the new files have an older timestamp than the prior > version that is stuck in the cache. > > The rsync process that I describe has not changed and has been in use for > many years. > > > -----Original Message----- > > From: Jason Breitman > > Sent: Thursday, August 25, 2022 11:54 AM > > To: Ben Hutchings <b...@decadent.org.uk>; 1017...@bugs.debian.org > > Subject: RE: Bug#1017720: nfs-common: No such file or directory > > > > I have the same issue after adding actimeo=30 to /etc/fstab, rebooting and > > testing. > > I also confirmed that those settings applied via /proc/mounts which shows > > the below snippet for each mountpoint. > > nfs4 > > > rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,acregmin=30,a > > > cregmax=30,acdirmax=30,hard,noresvport,proto=tcp,timeo=600,retrans=2,s > > > ec=krb5,clientaddr=X.X.X.X,lookupcache=pos,local_lock=none,addr=Y.Y.Y.Y 0 > > 0 > > > > > -----Original Message----- > > > From: Jason Breitman > > > Sent: Tuesday, August 23, 2022 2:42 PM > > > To: Ben Hutchings <b...@decadent.org.uk>; 1017...@bugs.debian.org > > > Subject: RE: Bug#1017720: nfs-common: No such file or directory > > > > > > What additional information can I provide for us to move forward with > this > > > process? > > > > > > To summarize and include further details, rsync is used to sync > applications > > to > > > a file server which behaves like a repository. > > > We do preserve timestamps from the build server and also use --delete. > > We > > > do not run the applications from the file server. All servers use NTP. > > > > > > The application has a sub-directory that contain files with version > numbers. > > > These are libraries. > > > When a new build is complete, a developer pushes their updates via > rsync > > to > > > the file server / repository. > > > > > > I believe that the dentry cache thinks the "old" files exist and > > > generates a > > No > > > such file or directory error showing question marks for that files > attributes. > > > Dropping the dentry cache via echo 2 > /proc/sys/vm/drop_caches > > resolves > > > the issue. > > > > > > This behavior is not observed in Debian 10.8 with that distributions > > associated > > > kernel and packages. > > > > > > > -----Original Message----- > > > > From: Jason Breitman > > > > Sent: Friday, August 19, 2022 9:52 PM > > > > To: Ben Hutchings <b...@decadent.org.uk>; 1017...@bugs.debian.org > > > > Subject: RE: Bug#1017720: nfs-common: No such file or directory > > > > > > > > > -----Original Message----- > > > > > From: Ben Hutchings <b...@decadent.org.uk> > > > > > Sent: Friday, August 19, 2022 7:27 PM > > > > > To: Jason Breitman <jbreit...@tildenparkcapital.com>; > > > > > 1017...@bugs.debian.org > > > > > Subject: Re: Bug#1017720: nfs-common: No such file or directory > > > > > > > > > > Control: tag -1 moreinfo > > > > > > > > > > On Fri, 2022-08-19 at 13:16 +0000, Jason Breitman wrote: > > > > > > Package: nfs-common > > > > > > Version: 1:1.3.4-6 > > > > > > Severity: important > > > > > > > > > > > > Kernel: 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) > > x86_64 > > > > > > GNU/Linux > > > > > > > > > > > > -- Description > > > > > > After updating and or creating new files on our file server via > > > > > > rsync, we see many files report the error message below from > NFSv4 > > > > > > clients since upgrading from Debian 10.8 to Debian 11.4. > > > > > > Clearing the dentry cache resolves the issue right away. > > > > > > I am not sure that nfs-common is the package to blame, but > > > > > > listed > > > > > > it based on the bug submission recommendations. > > > > > > > > > > The NFS implementation is mostly in the kernel, so probably this issue > > > > > belongs there. But the kernel team is responsible for both packages. > > > > > > > > > > [...] > > > > > > -- Error message > > > > > > ls: cannot access 'filename': No such file or directory > > > > > > -????????? ? ? ? ? ? filename > > > > > [...] > > > > > > > > > > So we know the file's there but can't stat it. I think this means the > > > > > client has cached the handle of the old file of that name, which has > > > > > been deleted. > > > > > > > > > > - Are client and server clocks closely synchronised? If not, that > > > > > needs to be fixed. > > > > > > > > > The clocks are synchronized using NTP. > > > > > > > > > - Are clients likely to read this directory while rsync is running, or > > > > > shortly before? If so, it may help to reduce the attribute caching > > > > > timeout on the client. See the "Directory entry caching" section in > > > > > the nfs(5) manual page. > > > > > > > > > Clients are not likely to read this directory while rsync is running > > > > for the > > > > observed cases. That can happen in our environment, but not in this > > case. > > > > I am using the lookupcache=pos option. I tried noac, but the > > performance > > > > penalty was too much. Which option are you referring to and what > > setting > > > > do you recommend testing? > > > > > > > > > I don't know why you're only seeing this after an upgrade of the > > > > > clients, though. I'm not aware that there has been any big change to > > > > > attribute caching. > > > > > > > > > I appreciate you responding to my report and am happy to answer any > > > > questions. > > > > We have multiple monitors and log scrapers to detect "file not found" > > > > exceptions that would let us know if this was happening before. > > > > To share more, I have 2 environments mounting from the same file > > server. > > > > Each environment has several servers. The issue is only seen in the > > > > environment running Debian 11.4. > > > > I also should have mentioned that the files in question have a version > > > > number appended. filename-1111. When the file is updated via rsync, > it > > is > > > > called filename-1112 and the prior file is removed. The error is about > > > > filename-1111. > > > > I am not sure if this is the proper terminology, but the issue appears > > > > to > be > > > > the negative dentry cache. > > > > > > > > > Ben. > > > > > > > > > > -- > > > > > Ben Hutchings > > > > > Beware of bugs in the above code; > > > > > I have only proved it correct, not tried it. - Donald Knuth > > > > > > > > Jason Breitman > > > Jason Breitman > > Jason Breitman > Jason Breitman Jason Breitman
