[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2025-01-22 Thread Silas Horton
As another data point, we're seeing this bug on one of our NFS servers running Ubuntu 24.04.1 with 6.8.0-51 kernel. Also experienced a similar issue to one of the above users with unable to start nfs-kernel-server and trying to restart the service. [2164406.282362] INFO: task nfsd:8039 blocked fo

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2025-01-20 Thread Gilbert Blom
Found this thread last year due to a new problem with NFS timeout errors on my server. That was after upgrade from 5.15.0-122-generic to 6.8.0-45-generic. Last week I upgraded to 6.8.0-51-generic. Immediately got NFS timeout errors during the builds. Now back again at 5.15.0-122-generic, which give

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2025-01-08 Thread Omen Wild
I see Ubuntu has released a 6.8.0-49.49~22.04.1 and a -50, and a -51. Has anyone updated to the newer kernel to see if it resolved this issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Tit

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-11-14 Thread Sambit Bikas Pal
Affecting me as well. Running 22.04.5 and 6.8.0-48-generic The common factor seems to be heavy file i/o over a 10gbps link. Didn't encounter this on 20.04, before the upgrade. [575448.138621] INFO: task nfsd:1760 blocked for more than 245 seconds. [575448.138623] Not tainted 6.8.0-48-gener

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-11-06 Thread Omen Wild
@mehmetbasaran: This issue is impacting one of the HPC clusters I manage. Are there any thoughts as to when this patch will get rolled out to a released 6.8.0 kernel? Also, does anyone know if this is a server bug or a client bug, i.e. for mitigation, do the servers need to be rolled back to a v

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-19 Thread Kraftzwerg
I push a large amount of Files (36TB) to a new Backup Server. This happened to me twice in the last 18 hours. Bug Description: nfs-server hangs after some hours all nfs-clients are affected. nfs-server can`t be stopped only power cycle helps --> DMESG nfs-server: [33547.322122] INFO: task nfsd:

AW: [Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-09 Thread Peter Schubert
r 2024 12:09 An: schub...@iap-kborn.de Betreff: [Bug 2062568] Re: nfsd gets unresponsive after some hours of operation Hi Peter, Thanks for trying out the nfs patch. However, "RPC: Could not send backchannel reply error: -110" I suspected this problem to be on the nfs server side rather

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-09 Thread Mehmet Basaran
Hi Peter, Thanks for trying out the nfs patch. However, "RPC: Could not send backchannel reply error: -110" I suspected this problem to be on the nfs server side rather than the nfs client. Were tou having this error on the client side before trying out the patched nfs kernel? "rcu: INFO: rcu_sch

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-09 Thread Mehmet Basaran
** Changed in: linux (Ubuntu) Assignee: Mehmet Basaran (mehmetbasaran) => Paolo Pisati (p-pisati) ** Changed in: linux (Ubuntu Noble) Assignee: Mehmet Basaran (mehmetbasaran) => Paolo Pisati (p-pisati) -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-08 Thread Andreas Hasenack
I'm marking the nfs-utils task as incomplete again, because so far this looks like a kernel issue, and not userspace. ** Changed in: nfs-utils (Ubuntu Noble) Status: Confirmed => Incomplete ** Changed in: nfs-utils (Ubuntu) Status: Confirmed => Incomplete -- You received this bug

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-07 Thread Launchpad Bug Tracker
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: nfs-utils (Ubuntu Noble) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Tit

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-10-04 Thread Kleber Sacilotto de Souza
** Also affects: nfs-utils (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Noble) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Noble) Status: New => In Progress ** Changed in: linux (Ubuntu Noble) Assignee: (unassi

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-30 Thread Peter Schubert
We installed the unofficial kernel 6.8.0-46-generic-nfs on several NFS client servers on Saturday and have been testing it with high IO loads since then. Unfortunately the server crashed again after about 40 hours with "rcu: INFO: rcu_sched self-detected stall on CPU". The kernel 6.8.0-46-generi

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-29 Thread Mehmet Basaran
If this patch proves a fix, we plan to release it in the next update. In this case, update will install the official kernel (which will also include this patch) and change grub settings to boot this kernel automatically. -- You received this bug notification because you are a member of Ubuntu Bug

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-27 Thread Kleber Sacilotto de Souza
** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title: n

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-26 Thread Robert Williams
Performed two upgrades from 22.04 yesterday, both have locked up overnight with the below. It feels like this is the same issue - can anyone confirm for me? kernel: INFO: task nfsd:2029 blocked for more than 122 seconds. kernel: Tainted: G OE 6.8.0-45-generic #45-Ubuntu kernel

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-25 Thread Mehmet Basaran
Hi all, I am from the Canonical's kernel team and currently investigating this issue. In this case, jammy-hwe, mantic-hwe, and noble by default uses 6.8 kernel (when a generic jammy and mantic is installed it uses hwe version by default). So, the issue is with 6.8 kernel rather than series. I was

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-25 Thread Mehmet Basaran
Additionally, for those who prefer to migrate to newer kernel you can try our mainline builds here: https://kernel.ubuntu.com/mainline/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title: n

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-20 Thread Mehmet Basaran
** Changed in: linux (Ubuntu) Assignee: (unassigned) => Mehmet Basaran (mehmetbasaran) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title: nfsd gets unresponsive after some hours of op

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-19 Thread DrewM
For those who can't switch to NFSv3 and don't want to run the .0 release 6.11.0 kernel from Oriole, there are 6.10 kernels from xanmod for Ubuntu https://xanmod.org/#apt_repository -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https:

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-18 Thread Launchpad Bug Tracker
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title: nfsd

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-18 Thread gbonazz...@bonaz.it
I also have this issue on Ubuntu 22.04.5 LTS with linux kernel tagged as 6.8.0-40-generic #40~22.04.3-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 30 17:30:19 UTC 2 NFSv3 workaround has serious consequences on all the clients that refused to downgrade the protocol. My only option, till a patch, is to dow

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-18 Thread Andreas Hasenack
Is anybody in a position to try out the kernel from the ubuntu 24.10 upcoming release? There will be a beta out this week of Ubuntu Oracular 24.10. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-18 Thread Andreas Hasenack
Thanks all for your input. I'll add a kernel task to this bug, but keep the userspace one open for now. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https:

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-09-18 Thread DrewM
I also have this issue and can't go more than about 8 hours without this breaking. This was not an issue in 20.04. Currently attempting the NFSv3 workaround There are mentions that this bug is fixed in kernel 6.9.8 https://lists.proxmox.com/pipermail/pve-devel/2024-July/064614.html -- You recei

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-08-31 Thread Stefan
Confirmed on 24.04.1 and previously on 23.10 (both server and client), also using large files (1-100GB) and 10G networking to large/fast disk arrays, which others have suggested to be a key factor. All mountpoints are running BTRFS (in some cases a brand new filesystem) without any LUKS. My observ

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-08-25 Thread Olle Liljenzin
Get the same in 22.04 now after 6.18 was rolled out as hwe kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title: nfsd gets unresponsive after some hours of operation To manage notifi

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-07-31 Thread Derek Harter
This is a me too. We encountered the same problem, with the exact same message of a tainted and hung nfsd task. Like yuhldr (yuh) we investigated and problem happens regularly. In our case, we have a smallish cluster (100 machines) with a gigabit ethernet switch network. The nfsd machine serves

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-06-27 Thread Ricardo Cruz
For the benefit of others... Like #10, we also have a cluster with this issue. As a workaround, we are using version 3 of the NFS protocol (`nfsvers=3` in `/etc/fstab`), which so far seems to have eliminated the problem for us. -- You received this bug notification because you are a member of Ubu

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-06-23 Thread yuhldr
I encountered the same problem. After several days of testing, the problem can be reproduced 100%. Ubuntu24.04, a 10Gb/s optical fiber connection is used between the login node and the computing node. The computing node uses nfs to mount the /home of the login node. The entire system is managed usi

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-06-17 Thread Jeff
The tricky part for me is that client was regularly changing, so I can't confidently say when did errors start appearing, it's just very suspicious that it needs high load (as new host and higher network bandwidth made the issue more frequent), and uploading to the server as just pure downloading d

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-06-17 Thread GuoqingJiang
IIUC, from nfs server side, both 23.10 (6.5 series) and 24.04 (6.8 series) have the similar issue, but 22.04 (probably 5.15.x kernel) was ok. And what is the kernel version from nfs client? Is it changed or stay on one certain version? Anyway, I guess the efficient way is to bisect which commit c

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-25 Thread Jeff
** Attachment added: "rpc_tasks.txt" https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2062568/+attachment/5770302/+files/rpc_tasks.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-25 Thread Jeff
Oh fun, trying to add multiple attachments, I just ended up finding a bug report from 2007 complaining about apport being able to do this, but the web interface is limited, so guess I'll get a bit spammy. Ran into the same as usual again, even with avoiding heavy utilization. Not seeing anything t

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-25 Thread Jeff
** Attachment added: "nfs_threads.txt" https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2062568/+attachment/5770301/+files/nfs_threads.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-22 Thread Jeff
Ran into this again just hours after commenting by attempting to unpack a large archive file. Apparently the new setup of higher performance host with more network bandwidth is just too overwhelming to be usable with NFS with this issue. Reading alone still seems to be fine though. Torturing 50

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-21 Thread Launchpad Bug Tracker
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: nfs-utils (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2062568 Title:

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-21 Thread Jeff
This kind of issue appeared with Ubuntu 23.10 for me on the server mostly using an HDD for bulk storage with a not exactly powerful CPU also being occupied with using WireGuard to secure the NFS connection. Mentioning the performance details because I have a feeling they matter. An also not exact

[Bug 2062568] Re: nfsd gets unresponsive after some hours of operation

2024-04-19 Thread Olle Liljenzin
** Description changed: - I installed the 22.04 Beta on two test machines that were running 22.04 + I installed the 24.04 Beta on two test machines that were running 22.04 without issues before. One of them exports two volumes that are mounted by the other machine, which primarily uses them as