On 06/27/2017 01:45 AM, Sahid Orentino Ferdjaoui wrote:
On Mon, Jun 26, 2017 at 12:12:49PM -0600, Chris Friesen wrote:
On 06/25/2017 02:09 AM, Sahid Orentino Ferdjaoui wrote:
On Fri, Jun 23, 2017 at 10:34:26AM -0600, Chris Friesen wrote:
On 06/23/2017 09:35 AM, Henning Schild wrote:
Am Fri, 23 Jun 2017 11:11:10 +0200
schrieb Sahid Orentino Ferdjaoui <sferd...@redhat.com>:
In Linux RT context, and as you mentioned, the non-RT vCPU can acquire
some guest kernel lock, then be pre-empted by emulator thread while
holding this lock. This situation blocks RT vCPUs from doing its
work. So that is why we have implemented [2]. For DPDK I don't think
we have such problems because it's running in userland.
So for DPDK context I think we could have a mask like we have for RT
and basically considering vCPU0 to handle best effort works (emulator
threads, SSH...). I think it's the current pattern used by DPDK users.
DPDK is just a library and one can imagine an application that has
cross-core communication/synchronisation needs where the emulator
slowing down vpu0 will also slow down vcpu1. You DPDK application would
have to know which of its cores did not get a full pcpu.
I am not sure what the DPDK-example is doing in this discussion, would
that not just be cpu_policy=dedicated? I guess normal behaviour of
dedicated is that emulators and io happily share pCPUs with vCPUs and
you are looking for a way to restrict emulators/io to a subset of pCPUs
because you can live with some of them beeing not 100%.
Yes. A typical DPDK-using VM might look something like this:
vCPU0: non-realtime, housekeeping and I/O, handles all virtual interrupts
and "normal" linux stuff, emulator runs on same pCPU
vCPU1: realtime, runs in tight loop in userspace processing packets
vCPU2: realtime, runs in tight loop in userspace processing packets
vCPU3: realtime, runs in tight loop in userspace processing packets
In this context, vCPUs 1-3 don't really ever enter the kernel, and we've
offloaded as much kernel work as possible from them onto vCPU0. This works
pretty well with the current system.
For RT we have to isolate the emulator threads to an additional pCPU
per guests or as your are suggesting to a set of pCPUs for all the
guests running.
I think we should introduce a new option:
- hw:cpu_emulator_threads_mask=^1
If on 'nova.conf' - that mask will be applied to the set of all host
CPUs (vcpu_pin_set) to basically pack the emulator threads of all VMs
running here (useful for RT context).
That would allow modelling exactly what we need.
In nova.conf we are talking absolute known values, no need for a mask
and a set is much easier to read. Also using the same name does not
sound like a good idea.
And the name vcpu_pin_set clearly suggest what kind of load runs here,
if using a mask it should be called pin_set.
I agree with Henning.
In nova.conf we should just use a set, something like
"rt_emulator_vcpu_pin_set" which would be used for running the emulator/io
threads of *only* realtime instances.
I'm not agree with you, we have a set of pCPUs and we want to
substract some of them for the emulator threads. We need a mask. The
only set we need is to selection which pCPUs Nova can use
(vcpus_pin_set).
We may also want to have "rt_emulator_overcommit_ratio" to control how many
threads/instances we allow per pCPU.
Not really sure to have understand this point? If it is to indicate
that for a pCPU isolated we want X guest emulator threads, the same
behavior is achieved by the mask. A host for realtime is dedicated for
realtime, no overcommitment and the operators know the number of host
CPUs, they can easily deduct a ratio and so the corresponding mask.
Suppose I have a host with 64 CPUs. I reserve three for host overhead and
networking, leaving 61 for instances. If I have instances with one non-RT
vCPU and one RT vCPU then I can run 30 instances. If instead my instances
have one non-RT and 5 RT vCPUs then I can run 12 instances. If I put all of
my emulator threads on the same pCPU, it might make a difference whether I
put 30 sets of emulator threads or 12 sets.
Oh I understand your point now, but not sure that is going to make any
difference. I would say the load in the isolated cores is probably
going to be the same. Even that an overhead will be the number of
threads handled which will be slightly higher in your first scenario.
The proposed "rt_emulator_overcommit_ratio" would simply say "nova is
allowed to run X instances worth of emulator threads on each pCPU in
"rt_emulator_vcpu_pin_set". If we've hit that threshold, then no more RT
instances are allowed to schedule on this compute node (but non-RT instances
would still be allowed).
Also I don't think we want to schedule where the emulator threads of
the guests should be pinned on the isolated cores. We will let them
float on the set of cores isolated. If there is a requierement to have
them pinned so probably the current implementation will be enough.
Once you use "isolcpus" on the host, the host scheduler won't "float" threads
between the CPUs based on load. To get the float behaviour you'd have to not
isolate the pCPUs that will be used for emulator threads, but then you run the
risk of the host running other work on those pCPUs (unless you use cpusets or
something to isolate the host work to a subset of non-isolcpus pCPUs).
Chris
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
