All,
We have a few clusters up and running now, being built by Crowbar 1.6.1
(mesa branch, build 3860, w/ Dell specific BIOS/RAID barclamps). The
clusters have all been performing extremely poorly from a nova-compute
standpoint. We resolved the issue, and would like to share how we
resolved that issue, in case anyone else runs in to this issue.
Our cluster has 2 nova-compute nodes. On those nodes, we instantiated a
VM Guest image (in this case a simple Fedora 19 guest). After eventually
being able to log in (it took around 10 minutes to instantiate/boot up,
and be available on the network after cloud-init ranŠ), we did a quick
benchmark of the CPU performance. First, we also did it on the ³host
node² compute node itself (eg at the Ubuntu O.S. level on the
Crowbar/OpenStack Compute node). For the CPU test, we simply used:
openssl speed rsa
At the host node level, we got:
root@db8-ca-3a-5e-d9-38:~# openssl speed rsa
<ŠsnipŠ>
sign verify sign/s verify/s
rsa 512 bits 0.000064s 0.000005s 15512.5 205154.6
The important take-away number here is ³205,154². Then, we logged in to
the Guest VM instance, and performed the same thing at Fedora CLI, and got:
[fedora@test-fedora ~]$ openssl speed rsa
<ŠsnipŠ>
sign verify sign/s verify/s
rsa 512 bits 0.002335s 0.000151s 428.2 6605.7
Take-away number ³6,605². BAD! BAD! VERY BAD!
Looking in to it - we noticed that the nova-compute instance was set up to
use QEMU for virtualization. BUT - we _did_ set the barclamp to use KVM.
Here¹s proof of the inconsistency between the Crowbar ³nova² proposal, and
the actual built up nova-compute node setting that was implemented:
delta-admin # crowbar nova proposal show proposal | grep libvirt_type
"libvirt_type": "kvm",
delta-admin # ssh dcn4 "grep libvirt_type /etc/nova/nova.conf"
libvirt_type=qemu
So to fix it, we did the following (we also set ³libvirt_cpu_type² from
³none² to ³host-model² at the same time):
for F in /opt/dell/chef/cookbooks/nova/templates/default/nova.conf.erb
/opt/dell/barclamps/nova/chef/cookbooks/nova/templates/default/nova.conf.er
b
do
sed -i.bak 's/^\(libvirt_type\)=qemu$/\1=kvm/;
s/^\(libvirt_cpu_mode\)=none/\1=host-model/' $F
cp $F.bak /root/
done
cd /opt/dell/barclamps/nova/chef/cookbooks/
knife cookbook upload nova -o .
# our two compute nodes
ssh dcn4 chef-client | tee -a /tmp/dcn4.log
ssh dcn5 chef-client | tee -a /tmp/dcn5.log
# reboot 'em for good measure
crowbar machines reboot `host -t CNAME dcn4 | awk ' { print $NF } ' |
sed
's/.\$//'`
crowbar machines reboot `host -t CNAME dcn5 | awk ' { print $NF } ' |
sed
's/.\$//'`
After these changes, we instantiated a new Guest VM, and performed the
same CPU benchmark. We now get:
[fedora@fedora-demo-1 ~]$ openssl speed rsa
<ŠsnipŠ>
sign verify sign/s verify/s
rsa 512 bits 0.000064s 0.000005s 15641.1 204646.2
³204,646² Yay! That¹s what we¹re talking about!!!!!
~~shane
_______________________________________________
Crowbar mailing list
Crowbar@dell.com
https://lists.us.dell.com/mailman/listinfo/crowbar
For more information: http://crowbar.github.com/
