Hi Roman,
I haven't but I will write a blueprint for the core pinning part.
I considered vcpu element usage as well but in that case you can not set
e.g. vcpu-0 to run on pcpu-0. Vcpus and emulator are sharing all pcpus
defined in cpuset so I decided to use cputune element.
Are you using extra specs for carrying cpuset attributes in your
implementation?
Br,Tuomas
On 18.11.2013 17:14, Roman Verchikov wrote:
Tuomas,
Have you published your code/blueprints anywhere? Looks like we’re working on
the same stuff. I have implemented almost the same feature set (haven’t
published anything yet because of this thread), except for the scheduler part.
The main goal is to be able to pin VCPUs in NUMA environment.
Have you considered adding placement and cpuset attributes to <vcpu> element?
For example:
<vcpu placement=‘static’ cpuset=‘%whatever%’>
Thanks,
Roman
On Nov 13, 2013, at 14:46, Tuomas Paappanen <tuomas.paappa...@tieto.com> wrote:
Hi all,
I would like to hear your thoughts about core pinning in Openstack. Currently
nova(with qemu-kvm) supports usage of cpu set of PCPUs what can be used by
instances. I didn't find blueprint, but I think this feature is for isolate
cpus used by host from cpus used by instances(VCPUs).
But, from performance point of view it is better to exclusively dedicate PCPUs
for VCPUs and emulator. In some cases you may want to guarantee that only one
instance(and its VCPUs) is using certain PCPUs. By using core pinning you can
optimize instance performance based on e.g. cache sharing, NUMA topology,
interrupt handling, pci pass through(SR-IOV) in multi socket hosts etc.
We have already implemented feature like this(PoC with limitations) to Nova
Grizzly version and would like to hear your opinion about it.
The current implementation consists of three main parts:
- Definition of pcpu-vcpu maps for instances and instance spawning
- (optional) Compute resource and capability advertising including free pcpus
and NUMA topology.
- (optional) Scheduling based on free cpus and NUMA topology.
The implementation is quite simple:
(additional/optional parts)
Nova-computes are advertising free pcpus and NUMA topology in same manner than
host capabilities. Instances are scheduled based on this information.
(core pinning)
admin can set PCPUs for VCPUs and for emulator process, or select NUMA cell for
instance vcpus, by adding key:value pairs to flavor's extra specs.
EXAMPLE:
instance has 4 vcpus
<key>:<value>
vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
emulator:5 --> emulator pinned to pcpu5
or
numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
In nova-compute, core pinning information is read from extra specs and added to
domain xml same way as cpu quota values(cputune).
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='3'/>
<vcpupin vcpu='3' cpuset='4'/>
<emulatorpin cpuset='5'/>
</cputune>
What do you think? Implementation alternatives? Is this worth of blueprint? All
related comments are welcome!
Regards,
Tuomas
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
