You have been subscribed to a public bug:
Ubuntu 10.04 LTS
Kernel 2.6.32-24-server
latest updates applied
libvirt-bin 0.7.5-5ubuntu27
qemu-kvm 0.12.3+noroms-0ubuntu9.2
Hardware
HP DL360 G6 server, 2 quad core Xeons E5520, 24GB RAM
Actual Nehalem CPUs are using the NUMA architecture to allocate RAM per CPU.
I did a lot of benchmarks with KVM on these machines, and Ubuntu shows a very
slow network throughput and i/o performance.
We suspect, that this has to do with libvirtd not recognizing the NUMA feature.
A simple "virsh capabilities" proves this:
<host>
<cpu>
<arch>x86_64</arch>
<model>core2duo</model>
<topology sockets='4' cores='4' threads='1'/>
<feature name='lahf_lm'/>
<feature name='rdtscp'/>
<feature name='popcnt'/>
<feature name='dca'/>
<feature name='xtpr'/>
<feature name='cx16'/>
<feature name='tm2'/>
<feature name='est'/>
<feature name='vmx'/>
<feature name='ds_cpl'/>
<feature name='pbe'/>
<feature name='tm'/>
<feature name='ht'/>
<feature name='ss'/>
<feature name='acpi'/>
<feature name='ds'/>
</cpu>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
</uri_transports>
</migration_features>
</host>
Expected result would be (RedHat and CentOS on the same machine)
<cpu>
<arch>x86_64</arch>
<model>core2duo</model>
<topology sockets='2' cores='4' threads='2'/>
--snip--
<topology>
<cells num='2'>
<cell id='0'>
<cpus num='8'>
<cpu id='0'/>
<cpu id='2'/>
<cpu id='4'/>
<cpu id='6'/>
<cpu id='8'/>
<cpu id='10'/>
<cpu id='12'/>
<cpu id='14'/>
</cpus>
</cell>
<cell id='1'>
<cpus num='8'>
<cpu id='1'/>
<cpu id='3'/>
<cpu id='5'/>
<cpu id='7'/>
<cpu id='9'/>
<cpu id='11'/>
<cpu id='13'/>
<cpu id='15'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<model>selinux</model>
<doi>0</doi>
</secmodel>
</host>
numactl --hardware:
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14
node 0 size: 12277 MB
node 0 free: 11659 MB
node 1 cpus: 1 3 5 7 9 11 13 15
node 1 size: 12287 MB
node 1 free: 11677 MB
node distances:
node 0 1
0: 10 20
1: 20 10
numastat shows a lot of misses, when doing benchmarks in a VM:
node0 node1
numa_hit 15527158 17015505
numa_miss 7032982 3512950
numa_foreign 3512950 7032982
interleave_hit 8078 8264
local_node 15525187 17006655
other_node 7034953 3521800
Also RedHat and CentOS perform a lot better.
When I get about 40-50MB/s network throughput (copying a file to an attached
iSCSI RAID) with Ubuntu, I get 120MB/s with CentOS.
Overall performance shows the same results and differences between the systems.
This is a very big problem, which makes KVM unusable.
We had to switch to CentOS 5.5 and Xen3, which performs even better than KVM on
CentOS.
But that's not a satisfying solution, because we rely just on Ubuntu for years
now.
Summary
Ubuntu libvirt doesn't recognize NUMA features and shows a very slow KVM
performance.
Other systems like CentOS recognize NUMA and perform a lot better.
Please fix it.
Additional info:
This posting (based on the experiences on our systems) gives a more detailed
description of this case:
http://marc.info/?l=kvm&m=127921447903105&w=2
cat /proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
stepping : 5
cpu MHz : 2266.449
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf
pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2
popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 4532.89
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
and so on until core 15.
Cheers,
Frank
** Affects: libvirt (Ubuntu)
Importance: Undecided
Status: New
** Tags: kvm libvirt numa
--
libvirt not recognizing NUMA architecture
https://bugs.launchpad.net/bugs/614322
You received this bug notification because you are a member of Ubuntu Server
Team, which is subscribed to libvirt in ubuntu.
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
