[Bug 249871] NFSv4 faulty directory listings under heavy load

2021-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #21 from Rick Macklem --- Good news. Hopefully it won't show up again. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing

[Bug 249871] NFSv4 faulty directory listings under heavy load

2021-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 Jason W. Bacon changed: What|Removed |Added Resolution|--- |Overcome By Events St

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-24 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #19 from Rick Macklem --- Well, I was going to suggest trying the "readahead=0" mount option on a client (which disabled read ahead, including for directory blocks). Then I discovered the code didn't allow that. So, first you'll

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-24 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #18 from Rick Macklem --- Created attachment 219047 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=219047&action=edit allow readahead=0 NFS mount option For some reason, the NFS client code did not allow readahead=0

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-24 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #17 from Jason W. Bacon --- (In reply to Rick Macklem from comment #13) rdir.c isn't reporting any problems on the server at the same time the NFS client is malfunctioning badly... On the server: FreeBSD login.peregrine baco

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-24 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #16 from Jason W. Bacon --- (In reply to Rick Macklem from comment #11) I created an NFS share on a compute node using UFS and the problem *DOES* occur there as well. Running the "ls" test from any NFS client to that node give

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-22 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #15 from Rick Macklem --- Sure. No rush. If the "Not monotonically increasing" prints out, you can just comment out that bit of code and run it again, although I think it is worth noting if/when it happens. (It should not break

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-22 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #14 from Jason W. Bacon --- (In reply to Rick Macklem from comment #13) Absolutely. I'm a little swamped at the moment, but will give this a try ASAP. Stand by... -- You are receiving this mail because: You are the assignee

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-21 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #13 from Rick Macklem --- I've put a small program called rdir.c in the attachments. You could run this program on the directory (locally on the NFS server) as follows: # rdir -v It should print out a bunch of numbers, which ar

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-21 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 Rick Macklem changed: What|Removed |Added Attachment #218960|text/x-csrc |text/plain mime type|

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-08 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #11 from Rick Macklem --- If it were possible for you to set up a server using UFS instead of ZFS, that would be nice, in that would tell us if the problem is in ZFS or more generic code. -- You are receiving this mail because

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-08 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #10 from Jason W. Bacon --- I suspect that listing directories like this is infrequently done and even less frequently verified. I don't recall ever doing something like this before that depended on an accurate directory listin

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #9 from Rick Macklem --- Well, if you can see the problem with vanilla NFSv3 (without rdirplus), then forget about the patch that prints out an error return for VOP_LOOKUP(). --> It in never called for NFSv3 Readdir RPCs. I hav

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #8 from Jason W. Bacon --- I thought you were onto something with the rdirplus hypothesis, but after a few more trials, I'm seeing some inconsistency with NFSv3 even without rdirplus. So we can scrap the notion that the proble

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-07 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #7 from Jason W. Bacon --- Doesn't appear to be an mbuf issue. Running the two scripts below simultaneously: #!/bin/sh -e cd AD-VCFs-whi/MAF-0.01-1000nt while true; do ls | grep 'combined.*\.vcf\.xz' | wc -l sleep 2

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-04 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #6 from Rick Macklem --- One additional thing you could do is: # vmstat -z | fgrep mbuf_cluster on the server when it is under heavy load, to see if it is running out of mbuf clusters. (Theoretically the NFS server should keep

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-04 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #5 from Jason W. Bacon --- Disabling debug.vfscache didn't help. Adding the patch will take a little more doing, as this is a production system. I'll try reproducing the problem on a private test cluster first, maybe later thi

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-02 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #4 from Rick Macklem --- Another thing to try is disabling the name cache. I think the sysctl is called debug.vfscache? (For some reason it has disappeared from head/current, but should be in 12.1.) --> Set it to 0 on the serve

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-02 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #3 from Rick Macklem --- Created attachment 218479 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=218479&action=edit log readdirplus VOP_LOOKUP() failures This little patch adds a printf() that logs VOP_LOOKUP() erro

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-10-01 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #2 from Jason W. Bacon --- Thanks for the suggestions. I disabled intr and I'm still getting variable results. Server and clients are all FreeBSD 12.1 and I am not using nfsuserd. Server: vfs.zfs.min_auto_ashift=12 net.inet.

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-09-26 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 --- Comment #1 from Rick Macklem --- Yep, that is weird. Assuming the clients are FreeBSD and not Linux, the only thing I can think of to try is take the "intr" option off the mounts. The BUGS section of "man mount_nfs" notes it should neve

[Bug 249871] NFSv4 faulty directory listings under heavy load

2020-09-24 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249871 Bug ID: 249871 Summary: NFSv4 faulty directory listings under heavy load Product: Base System Version: 12.1-RELEASE Hardware: amd64 OS: Any Status: New