Hi Salil,
On 8/21/24 8:23 PM, Salil Mehta wrote:
On 8/21/24 2:40 AM, Salil Mehta wrote:
>
> I donβt understand this clearly. Are you suggesting to reuse only
> single vCPU object to initialize all KVM vCPUs not yet plugged? If
> yes, then I'm not sure what do we gain here by adding this complexity?
> It does not consume time or resources because we are not realizing any
> of these vCPU object in any case.
>
First of all, it seems we have different names and terms for those cold-
booted vCPUs and hotpluggable vCPUs. For example, vCPU-0 and vCPU-1
are cold-booted vCPUs while
vCPU-2 and vCPU-3 are hotpluggable vCPUs when we have '-smp
maxcpus=4,cpus=2'. Lets stick to convention and terms for easier discussion.
The idea is avoid instantiating hotpluggable vCPUs in virtmach_init() and
released in the same function for those hotpluggable vCPUs. As I can
understand, those hotpluggable vCPU instances are serving for two
purposes: (1) Relax the constraint that all vCPU's (kvm) file descriptor have
to be created and populated;
We are devising *workarounds* in Qemu for the ARM CPU architectural constraints
in KVM and in Guest Kernel, *not relaxing* them. We are not allowed to meddle
with
the constraints. That is the whole point.
Not having to respect those constraints led to rejection of the earlier
attempts to
upstream Virtual CPU Hotplug for ARM.
I meant to 'overcome' the constraints by 'relax'. My apologies if there're any
caused
confusions. Previously, you had attempt to create all vCPU objects and reuse
them when
vCPU hot added. In current implementation, the hotpluggable vCPUs are
instantiated and
released pretty soon. I was bringing the third possibility, to avoid
instantiating those
hotpluggable vCPU objects, for discussion. In this series, the life cycle of
those
hotpluggable vCPU objects are really short. Again, I didn't say we must avoid
instantiating
those vCPU objects, I brought the topic ONLY for discussion.
(2) Help to instantiate and realize
GICv3 object.
For (1), I don't think we have to instantiate those hotpluggable vCPUs at all.
In the above example where we have command line '-smp
maxcpus=4,cpus=2', it's unnecessary to instantiate
vCPU-3 and vCPU-4 to create and populate their KVM file descriptors.
We cannot defer create vCPU in KVM after GIC has been initialized in KVM.
It needs to know every vCPU that will ever exists right at the time it is
getting
Initialized. This is an ARM CPU Architectural constraint.
It will be appreciated if more details other than 'an ARM CPU architectural
constraint'
can be provided. I don't understand this constrain very well at least.
A
vCPU's KVM file descriptor is create and populated by the following ioctls
and function calls. When the first vCPU (vCPU-0) is realized, the property
corresponding to "&init" is fixed for all vCPUs. It means all vCPUs have same
properties except the "vcpu_index".
ioctl(vm-fd, KVM_CREATE_VCPU, vcpu_index);
ioctl(vcpu-fd, KVM_ARM_VCPU_INIT, &init);
kvm_park_vcpu(cs);
A vCPU's properties are determined by two sources and both are global. It
means all vCPUs should have same properties: (a) Feature registers
returned from the host. The function
kvm_arm_get_host_cpu_features() is called for once, meaning this source
is same to all vCPUs;
Sure, but what are you trying to save here?
As mentioned above, the life cycle of those hotpluggable vCPU objects are really
short. They still consume time and memory to instantiate them. If I'm correct,
one of the primary goal for vCPU hotplug feature is to save system boot-up time,
correct?
(b) The parameters provided by user through '-cpu host,sve=off' are
translated to global properties and applied to all vCPUs when they're
instantiated.
Sure. Same is the case with PMU and other per-vCPU parameters.
We do not support heterogenous computing and therefore we do not
have per-vCPU control of these features as of now.
(a) (b)
aarch64_host_initfn qemu_init
kvm_arm_set_cpu_features_from_host parse_cpu_option
kvm_arm_get_host_cpu_features cpu_common_parse_features
qdev_prop_register_global
:
device_post_init
qdev_prop_set_globals
Sure, I understand the code flow but what are you trying to suggest here?
I tried to explain that vCPU object isn't needed to create and populate vCPU's
file
descriptors, as highlight in (1). The information used to create the cold-booted
vCPU-0 can be reused because all vCPUs have same properties and feature set.
For (2), I'm still looking into the GICv3 code for better understanding.
Oh, I thought you said you've finished your reviews π
Please take your time. For your reference, you might want to check:
KVMForum 2023:
https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf
https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf
KVMForum 2020:
https://kvm-forum.qemu.org/2020/Oct%2029_Challenges%20in%20Supporting%20Virtual%20CPU%20Hotplug%20in%20SoC%20Based%20Systems%20like%20ARM64_Salil%20Mehta.pdf
hmm, 'finished my review' has been misread frankly. By that, I meant I finished
my tests and
provided all the comments I had. Some of them are questions and discussions,
which I still need
to follow up.
Until
now, I don't see we need the instantiated hotpluggable vCPUs either.
I think, I've already answered this above it is because of ARM Architectural
constraint.
For
example, the redistributor regions can be exposed based on 'maxcpus'
instead of 'cpus'.
You mean during the review of the code you found that we are not doing it?
It's all about the discussion to the possibility to avoid instantiating
hotpluggable
vCPU objects.
The IRQ connection and teardown can be dynamically
done by connecting the board with GICv3 through callbacks in
ARMGICv3CommonClass.
The connection between GICv3CPUState and CPUARMState also can be
done dynamically.
Are you suggesting this after reviewing the code or you have to review it yet? π
I was actually trying to ask for your input and feedback. I was hoping your
input
to clear my puzzles: why vCPU objects must be in place to create GICv3 object?
Is it possible to create the GICv3 object without those vCPU objects? What kinds
of efforts we need to avoid instantiating those hotpluggable vCPU objects.
The best way perhaps is to find the answer from the code by myself ;-)
Thanks,
Gavin
