I push a large amount of Files (36TB) to a new Backup Server.
This happened to me twice in the last 18 hours.
Bug Description:
nfs-server hangs after some hours
all nfs-clients are affected.
nfs-server can`t be stopped
only power cycle helps
--> DMESG nfs-server:
[33547.322122] INFO: task nfsd:1020 blocked for more than 122 seconds.
[33547.328449] Tainted: G C E 6.8.0-1013-raspi #14-Ubuntu
[33547.335379] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[33547.343255] task:nfsd state:D stack:0 pid:1020 tgid:1020
ppid:2 flags:0x00000008
[33547.343264] Call trace:
[33547.343266] __switch_to+0xb8/0xd8
[33547.343275] __schedule+0x2f0/0x8a0
[33547.343280] schedule+0x3c/0x138
[33547.343284] schedule_timeout+0x1b0/0x1d0
[33547.343290] wait_for_completion+0xcc/0x178
[33547.343294] __flush_workqueue+0x110/0x410
[33547.343300] nfsd4_probe_callback_sync+0x24/0x38 [nfsd]
[33547.343371] nfsd4_destroy_session+0x168/0x228 [nfsd]
[33547.343436] nfsd4_proc_compound+0x4c0/0x770 [nfsd]
[33547.343502] nfsd_dispatch+0xc8/0x278 [nfsd]
[33547.343573] svc_process_common+0x44c/0x720 [sunrpc]
[33547.343670] svc_process+0xec/0x168 [sunrpc]
[33547.343760] svc_handle_xprt+0x3e0/0x5f0 [sunrpc]
[33547.343850] svc_recv+0x17c/0x338 [sunrpc]
[33547.343939] nfsd+0xc0/0x1d0 [nfsd]
[33547.344010] kthread+0xf4/0x108
[33547.344015] ret_from_fork+0x10/0x20
--> DMESG nfs-client:
[52581.872230] RPC: Could not send backchannel reply error: -110
--> systemctl status nfs-kernel-server after attemting to stop service:
× nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled;
preset: enabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: failed (Result: timeout) since Sat 2024-10-19 11:17:06 CEST; 11min
ago
Duration: 11h 11min 55.954s
Main PID: 1000 (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 9375)
Memory: 336.0K (peak: 492.0K)
CPU: 4ms
CGroup: /system.slice/nfs-server.service
├─8086 /usr/sbin/rpc.nfsd 0
└─8107 /usr/sbin/exportfs -au
systemd[1]: nfs-server.service: Processes still around after SIGKILL. Ignoring.
systemd[1]: nfs-server.service: State 'stop-post' timed out. Terminating.
systemd[1]: nfs-server.service: State 'final-sigterm' timed out. Killing.
systemd[1]: nfs-server.service: Killing process 8107 (exportfs) with signal
SIGKILL.
systemd[1]: nfs-server.service: Killing process 8086 (rpc.nfsd) with signal
SIGKILL.
systemd[1]: nfs-server.service: Processes still around after final SIGKILL.
Entering failed mode.
systemd[1]: nfs-server.service: Failed with result 'timeout'.
systemd[1]: nfs-server.service: Unit process 8086 (rpc.nfsd) remains running
after unit stopped.
systemd[1]: nfs-server.service: Unit process 8107 (exportfs) remains running
after unit stopped.
systemd[1]: Stopped nfs-server.service - NFS server and services.
Specs:
Raspberry Pi 5 8GB
Ubuntu 24.04.1 LTS
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2062568
Title:
nfsd gets unresponsive after some hours of operation
Status in linux package in Ubuntu:
In Progress
Status in nfs-utils package in Ubuntu:
Incomplete
Status in linux source package in Noble:
In Progress
Status in nfs-utils source package in Noble:
Incomplete
Bug description:
I installed the 24.04 Beta on two test machines that were running
22.04 without issues before. One of them exports two volumes that are
mounted by the other machine, which primarily uses them as a secondary
storage for ccache.
After being up for a couple of hours (happened twice since yesterday
evening) it seems that nfsd on the machine exporting the volumes hangs
on something.
From dmesg on the server (repeated a few times):
[11183.290548] INFO: task nfsd:1419 blocked for more than 1228 seconds.
[11183.290558] Not tainted 6.8.0-22-generic #22-Ubuntu
[11183.290563] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[11183.290582] task:nfsd state:D stack:0 pid:1419 tgid:1419
ppid:2 flags:0x00004000
[11183.290587] Call Trace:
[11183.290602] <TASK>
[11183.290606] __schedule+0x27c/0x6b0
[11183.290612] schedule+0x33/0x110
[11183.290615] schedule_timeout+0x157/0x170
[11183.290619] wait_for_completion+0x88/0x150
[11183.290623] __flush_workqueue+0x140/0x3e0
[11183.290629] nfsd4_probe_callback_sync+0x1a/0x30 [nfsd]
[11183.290689] nfsd4_destroy_session+0x186/0x260 [nfsd]
[11183.290744] nfsd4_proc_compound+0x3af/0x770 [nfsd]
[11183.290798] nfsd_dispatch+0xd4/0x220 [nfsd]
[11183.290851] svc_process_common+0x44d/0x710 [sunrpc]
[11183.290924] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[11183.290976] svc_process+0x132/0x1b0 [sunrpc]
[11183.291041] svc_handle_xprt+0x4d3/0x5d0 [sunrpc]
[11183.291105] svc_recv+0x18b/0x2e0 [sunrpc]
[11183.291168] ? __pfx_nfsd+0x10/0x10 [nfsd]
[11183.291220] nfsd+0x8b/0xe0 [nfsd]
[11183.291270] kthread+0xef/0x120
[11183.291274] ? __pfx_kthread+0x10/0x10
[11183.291276] ret_from_fork+0x44/0x70
[11183.291279] ? __pfx_kthread+0x10/0x10
[11183.291281] ret_from_fork_asm+0x1b/0x30
[11183.291286] </TASK>
From dmesg on the client (repeated a number of times):
[ 6596.911785] RPC: Could not send backchannel reply error: -110
[ 6596.972490] RPC: Could not send backchannel reply error: -110
[ 6837.281307] RPC: Could not send backchannel reply error: -110
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: nfs-kernel-server 1:2.6.4-3ubuntu5
ProcVersionSignature: Ubuntu 6.8.0-22.22-generic 6.8.1
Uname: Linux 6.8.0-22-generic x86_64
.etc.request-key.d.id_resolver.conf: create id_resolver * *
/usr/sbin/nfsidmap -t 600 %k %d
ApportVersion: 2.28.1-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: pass
Date: Fri Apr 19 14:10:25 2024
InstallationDate: Installed on 2024-04-16 (3 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Beta amd64
(20240410.1)
NFSMounts:
NFSv4Mounts:
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
XDG_RUNTIME_DIR=<set>
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2062568/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
