On Tue, Mar 24, 2020 at 10:11:15AM +0000, Tian, Kevin wrote:
> > From: Roger Pau Monné <roger....@citrix.com>
> > Sent: Tuesday, March 24, 2020 5:51 PM
> >
> > On Tue, Mar 24, 2020 at 06:03:28AM +0000, Tian, Kevin wrote:
> > > > From: Roger Pau Monne <roger....@citrix.com>
> > > > Sent: Saturday, March 21, 2020 3:08 AM
> > > >
> > > > The current usage of nvmx_update_apicv is not clear: it is deeply
> > > > intertwined with the Ack interrupt on exit VMEXIT control.
> > > >
> > > > The code in nvmx_update_apicv should update the SVI (in service
> > interrupt)
> > > > field of the guest interrupt status only when the Ack interrupt on
> > > > exit is set, as it is used to record that the interrupt being
> > > > serviced is signaled in a vmcs field, and hence hasn't been injected
> > > > as on native. It's important to record the current in service
> > > > interrupt on the guest interrupt status field, or else further
> > > > interrupts won't respect the priority of the in service one.
> > > >
> > > > While clarifying the usage make sure that the SVI is only updated when
> > > > Ack on exit is set and the nested vmcs interrupt info field is valid. Or
> > > > else a guest not using the Ack on exit feature would loose interrupts as
> > > > they would be signaled as being in service on the guest interrupt
> > > > status field but won't actually be recorded on the interrupt info vmcs
> > > > field, neither injected in any other way.
> > >
> > > It is insufficient. You also need to update RVI to enable virtual
> > > injection
> > > when Ack on exit is cleared.
> >
> > But RVI should be updated in vmx_intr_assist in that case, since
> > nvmx_intr_intercept shouldn't intercept the interrupt, as it should be
> > handled normally.
>
> As we discussed before, vmx_intr_assist is invoked before nvmx_switch_guest.
>
> It is incorrectly to update RVI at that time since it might be still vmcs02
> being
> active (if no pending softirq to make it invoked again).
>
> Also nvmx_intr_intercept does intercept Ack-on-exit=0 case:
>
> if ( intack.source == hvm_intsrc_pic ||
> intack.source == hvm_intsrc_lapic )
> {
> vmx_inject_extint(intack.vector, intack.source);
>
> ctrl = get_vvmcs(v, VM_EXIT_CONTROLS);
> if ( ctrl & VM_EXIT_ACK_INTR_ON_EXIT )
> {
> /* for now, duplicate the ack path in vmx_intr_assist */
> hvm_vcpu_ack_pending_irq(v, intack);
> pt_intr_post(v, intack);
>
> intack = hvm_vcpu_has_pending_irq(v);
> if ( unlikely(intack.source != hvm_intsrc_none) )
> vmx_enable_intr_window(v, intack);
> }
> else if ( !cpu_has_vmx_virtual_intr_delivery )
> vmx_enable_intr_window(v, intack);
>
> return 1; <<<<<<<<
Right, I always get confused by the switch happening in the vmentry
path.
That only happens when the vcpu is in nested mode
(nestedhvm_vcpu_in_guestmode == true). That would be the case before a
vmexit, at least for the first call to vmx_intr_assist if there are
pending softirqs.
Note that if there are pending softirqs the second time
nvmx_intr_intercept will return 0.
> }
>
> >
> > > >
> > > > Signed-off-by: Roger Pau Monné <roger....@citrix.com>
> > > > ---
> > > > xen/arch/x86/hvm/vmx/vvmx.c | 11 ++++++++++-
> > > > 1 file changed, 10 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/xen/arch/x86/hvm/vmx/vvmx.c
> > b/xen/arch/x86/hvm/vmx/vvmx.c
> > > > index 1b8461ba30..180d01e385 100644
> > > > --- a/xen/arch/x86/hvm/vmx/vvmx.c
> > > > +++ b/xen/arch/x86/hvm/vmx/vvmx.c
> > > > @@ -1383,7 +1383,7 @@ static void (struct vcpu *v)
> > > > {
> > > > struct nestedvmx *nvmx = &vcpu_2_nvmx(v);
> > > > unsigned long reason = get_vvmcs(v, VM_EXIT_REASON);
> > > > - uint32_t intr_info = nvmx->intr.intr_info;
> > > > + unsigned long intr_info = get_vvmcs(v, VM_EXIT_INTR_INFO);
> > >
> > > well, above reminds me an interesting question. Combining last and this
> > > patch, we'd see essentially that it goes back to the state before f96e1469
> > > (at least when Ack on exit is true).
> >
> > Well, patch 1/3 is a revert of f96e1469, so just reverting f96e1469
> > gets us to that state.
>
> you are right. I just wanted to point out that this patch alone doesn't
> do any real fix for ack-on-exit=1 case. It just makes sure that ack-on-exit=0
> is skipped from that function. So it won't be the one fixing your previous
> problem. 😊
Yes, that's correct.
> >
> > This patch is an attempt to clarify that nvmx_update_apicv is closely
> > related to the Ack on exit feature, as it modifies SVI in order to
> > signal the vector currently being serviced by the EXIT_INTR_INFO vmcs
> > field. This was not obvious to me, as at first sight I assumed
> > nvmx_update_apicv was actually injecting that vector into the guest.
> >
> > > iirc, that commit was introduced to enable
> > > nested x2apic with apicv, and your very first version even just removed
> > > the whole nvmx_update_apicv. Then now with the new reverted logic,
> > > are you still suffering x2apic problem? If not, does it imply the real fix
> > > is actually coming from patch 3/3 for eoi bitmap update?
> >
> > Yes, patch 3/3 is the one that fixed the issue. Note however that
> > strangely enough removing the call to nvmx_update_apicv (as my first
> > attempt to solve the issue) did fix it on one of my boxes. It depends
> > a lot on the available vmx features AFAICT.
>
> Did you confirm that with 3/3 alone can fix that issue? Just want to make
> sure the real gain of each patch, so we can reflect it in the commit msg
> in updated version.
Yes, the patch that actually fixes the issue in the box I've been
testing with is 3/3.
Xen will always use the Ack on exit feature, I currently have no way
to test whether not using Ack on exit works at all.
> >
> > > >
> > > > if ( reason == EXIT_REASON_EXTERNAL_INTERRUPT &&
> > > > nvmx->intr.source == hvm_intsrc_lapic &&
> > > > @@ -1399,6 +1399,15 @@ static void nvmx_update_apicv(struct vcpu *v)
> > > > ppr = vlapic_set_ppr(vlapic);
> > > > WARN_ON((ppr & 0xf0) != (vector & 0xf0));
> > > >
> > > > + /*
> > > > + * SVI must be updated when the interrupt has been signaled
> > > > using
> > the
> > > > + * Ack on exit feature, or else the currently in-service
> > > > interrupt
> > > > + * won't be respected.
> > > > + *
> > > > + * Note that this is specific to the fact that when doing a
> > > > VMEXIT an
> > > > + * interrupt might get delivered using the interrupt info vmcs
> > > > field
> > > > + * instead of being injected normally.
> > > > + */
> > >
> > > I'm not sure mentioning SVI specifically here is necessary, since all
> > > steps
> > > here are required - updating iir, isr, rvi, svi, ppr, etc. It is just an
> > > emulation
> > > of updating virtual APIC state as if a virtual interrupt delivery happens.
> >
> > Hm, it was hard for me to figure out that SVI is modified here in
> > order to signal that the Ack on exit vector is currently in service,
> > as opposed to being injected using the virtual interrupt delivery
> > mechanism.
> >
> > I just wanted to clarify that the purpose of this function is not to
> > inject the vector in intr_info, but rather to signal that such vector
> > has already been injected using a different mechanism.
> >
> > I'm happy to reword this, but IMO it should be clear that the purpose
> > of the function is not so much to inject an interrupt, but to update
> > the virtual interrupt delivery field in order to signal that an
> > interrupt has been injected using a different mechanism.
> >
>
> reading it again I feel possibly fine to put the comment there. But
> I disagree the statement above. The purpose of this function is indeed
> for injecting an interrupt. Both RVI and SVI are additional requirements
> when virtual APIC page is in-use while virtual interrupt delivery is not
> used, i.e. when ack-on-exit is set.
Right, so if I understand this correctly:
- SVI: must be updated when Ack on exit is used, to signal the
current in service interrupt which has been injected using a
mechanism different than virtual interrupts.
- RVI: should always be updated if there are pending interrupts to
be delivered. Note that the interrupt signaled in the INTR_INFO
field if Ack on exit is enabled is not considered pending anymore.
So I think RVI should always be updated, regardless of whether Ack on
exit is used or not. Do you agree?
Thanks, Roger.
