As this is a server system, there is no firefox or X window system on it.
Apport-collect starts a text-mode browser and in that I could authorize the
collecting of logs, it seems.
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
** Tags added: apport-collected
** Description changed:
Problem:
--------------------------------------------------------------
I just had to remove ubuntu server 12.04 to install redhat enterprise linux
6. The intermittent slowness was
completely unacceptable for the users, who have workstations with /home
mounted with nfs4 on this server.
The mail server, also accesses the /home because the /home/$USER/Maildir
directories are there.
Using nfs4, the kernel nfs threads caused enormous load.
The users had frozen desktops (greyed out windows) and mail slowed or
arrived days later as a result.
With RHEL6, all nfs4 problems are completely gone. I used the exact same
/etc/exports file, and the same
settings and mount options on the workstations, the same number of nfs
threads.
Both the redhat and ubuntu systems are KVM virtual guests on an redhat 6
virtual host (one of 3 actually).
The storage backend is a very fast equallogic array, which exports iscsi
targets to the virtual hosts.
I am sorry, but I have to conclude the current nfs4 implementation of ubuntu
server 12.04 is NOT fit for use.
A complete university department suffered for weeks while I tried to solve
the problems with ubuntu, but
in the end it was decided to install redhat instead, re-using the same iscsi
targets for system, home and data.
A missed chance for ubuntu...
Therefore I urge Canonical's people to classify this bug as critical.
Also I think quality assurance should have caught this bug before
shipping.
Analysis
------------------------------------------------------------
LOAD:
The nfs threads cause the kernel to use enormous amounts of 'sy' time as
measured in top.
I will attach a sample of top's output, of a particular _quiet_ time on the
network. Load is 7.82.
On busier moments, the load went through the roof, beyond 50 and further. It
consumes
actual CPU cycles.
Each thread consumes upto 30% of a cpu core. I enabled 128 threads.
rx an tx block sizes are 32768 on the clients. Both server and clients used
async, both on redhat and ubuntu.
SYSTEM vs IO-WAIT:
The replacement redhat system can surely be overloaded, but then it does not
consume CPU cycles doing so. Top does report high load, but it spends in in the
'wa' state. This indicates it is simply waiting for its backend iscsi devices
to complete writes. I tested this by simultaneously letting all workstations
write multi-gigabyte files with dd to /home.
On ubuntu, the nfs threads spend their time in 'sy', doing who-knows-what.
LOGS:
Nothing at all appears in the logs. But when I set bitwise debug options in
the /proc/sys/sunrpc/*debug files,
lots of log entries appear. Those seem like normal NFS protocol messages to
me though.
I also tried to discover what was happening with wireshark, but the traffic
looks like normal nfs4 traffic to me.
SLOWNESS:
That is the thing. The ubuntu nfs server is actually NOT slow at all. During
my dd tests, it wrote half a gigabyte per second to its iscsi backends. It's
_throughput_ is better than that of the redhat server.
As far as I can tell, it falls down because it makes client side processes
that want to do IO wait on other writes. A simple 'ls' has to wait until a
write has been completed. And both server and client used async nfs. People's
firefoxes freeze all the time because firefox need to read and write a lot to
its cache and other files in the .mozilla directory.
The dovecot imap server almost grinds to a halt trying to write all those
little files in people's /home/$USER/Maildir's.
The problems go on and on. Basically, a complete network of workstations is
almost unusable because of this.
Upfront tests were done of course, but showed only the excellent throughput
but not the appalling `waiting´ behaviour.
With redhat 6 there is no such problem.
SITUATION:
People use their own, and each others linux workstations for science, doing
large calculations and writing a lot of big and small files to nfs. The nfs
server serves /export/home, and also raw data storage from /export/data with
nfs4. The clients mount those under /home and /data/misc respectively. Also
there is a read-only software mount for certain scientific packages.
CONFIG:
client fstab lines:
#### nfs entries ###
sw.lorentz.leidenuniv.nl:/sw /sw nfs4
hard,intr,ro,tcp,rsize=32768,wsize=32768,bg,acl,async
home.lorentz.leidenuniv.nl:/home /home nfs4
hard,intr,rw,tcp,rsize=32768,wsize=32768,bg,acl,async
server exports file:
/export
132.229.227.0/24(ro,sync,insecure,root_squash,no_subtree_check,nohide,fsid=0)\
132.229.216.128/26(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.3(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)\
132.229.226.4(ro,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)
/export/home
132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.214.41(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/data
132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
/export/sw
132.229.227.0/24(rw,async,insecure,root_squash,no_subtree_check,nohide)\
132.229.216.128/26(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.3(rw,async,insecure,no_root_squash,no_subtree_check,nohide)\
132.229.226.4(rw,async,insecure,no_root_squash,no_subtree_check,nohide)
root@gaia:~# lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04
root@gaia2:~# dpkg -l | grep -E 'nfs|linux-image'
ii libnfsidmap2 0.25-1ubuntu2 NFS idmapping
library
ii linux-image-3.2.0-18-generic 3.2.0-18.29 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-19-generic 3.2.0-19.31 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-20-generic 3.2.0-20.33 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-21-generic 3.2.0-21.34 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-23-generic 3.2.0-23.36 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel
image for version 3.2.0 on 64 bit x86 SMP
ii linux-image-server 3.2.0.24.26 Linux kernel
image on Server Equipment.
ii nfs-common 1:1.2.5-3ubuntu3 NFS support
files common to client and server
ii nfs-kernel-server 1:1.2.5-3ubuntu3 support for
NFS kernel server
ii nfswatch 4.99.11-1 Program to
monitor NFS traffic for the console
WHAT I EXPECTED TO HAPPEN
----------------------------------
A fast and responsive nfs service.
WHAT HAPPENED INSTEAD
-----------------------------
I got fast, but also intermittently totally unresponsive.
+ ---
+ AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.0.1-0ubuntu7
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path',
'/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c',
'/dev/snd/pcmC0D0p', '/dev/snd/timer'] failed with exit code 1:
+ CRDA: Error: [Errno 2] No such file or directory
+ Card0.Amixer.info: Error: [Errno 2] No such file or directory
+ Card0.Amixer.values: Error: [Errno 2] No such file or directory
+ CurrentDmesg: [ 17.280028] eth0: no IPv6 routers present
+ DistroRelease: Ubuntu 12.04
+ HibernationDevice: RESUME=UUID=aafa9be4-19ac-4d74-a853-a5532ddedf5d
+ InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Alpha amd64
(20120313)
+ IwConfig:
+ lo no wireless extensions.
+
+ eth0 no wireless extensions.
+ Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+ MachineType: Red Hat KVM
+ NonfreeKernelModules: nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ext2
snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd psmouse serio_raw
lp parport i2c_piix4 soundcore snd_page_alloc virtio_balloon mac_hid floppy
+ Package: linux (not installed)
+ ProcEnviron:
+ LANGUAGE=en_US:en
+ TERM=xterm
+ PATH=(custom, no user)
+ LANG=en_US.UTF-8
+ SHELL=/bin/bash
+ ProcFB: 0 EFI VGA
+ ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-24-generic
root=/dev/mapper/gaia-root ro
+ ProcVersionSignature: Ubuntu 3.2.0-24.39-generic 3.2.16
+ RelatedPackageVersions:
+ linux-restricted-modules-3.2.0-24-generic N/A
+ linux-backports-modules-3.2.0-24-generic N/A
+ linux-firmware 1.79
+ RfKill: Error: [Errno 2] No such file or directory
+ Tags: precise
+ Uname: Linux 3.2.0-24-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups:
+
+ dmi.bios.date: 01/01/2007
+ dmi.bios.vendor: Seabios
+ dmi.bios.version: 0.5.1
+ dmi.chassis.type: 1
+ dmi.chassis.vendor: Red Hat
+ dmi.modalias:
dmi:bvnSeabios:bvr0.5.1:bd01/01/2007:svnRedHat:pnKVM:pvrRHEL6.2.0PC:cvnRedHat:ct1:cvr:
+ dmi.product.name: KVM
+ dmi.product.version: RHEL 6.2.0 PC
+ dmi.sys.vendor: Red Hat
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1006446
Title:
nfs4 causes enormous load in ubuntu-server making it unusable
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006446/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
