Coupling between KVM_IRQFD and KVM_SET_GSI_ROUTING?
Hello, I have a question related to KVM_IRQFD and KVM_SET_GSI_ROUTING ioctl relationship. When reading the KVM API documentation I do not understand there is any dependency between KVM_IRQFD and KVM_SET_GSI_ROUTING. According to the text it seems only the gsi field is used and interpreted as the irqchip pin. However irqchip.c kvm_set_irq code relies on an existing and not dummy routing table. My question is: does anyone agree on the fact the user-side must set a consistent routing table using KVM_SET_GSI_ROUTING before using KVM_IRQFD? The other alternative would have been to build a default identity GSI routing table in the kernel (gsi = irqchip.pin). In the positive, shouldn't we clarify the KVM API documentation? Thank you in advance Best Regards Eric -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] ARM: KVM: add irqfd and irq routing support
On 06/06/2014 02:16 PM, Christoffer Dall wrote: > On Mon, Jun 02, 2014 at 02:54:12PM +0100, Marc Zyngier wrote: >> Hi Eric, >> >> On Mon, Jun 02 2014 at 8:29:56 am BST, Eric Auger >> wrote: >>> This patch enables irqfd and irq routing on ARM. >>> >>> It turns on CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQ_ROUTING >>> >>> irqfd framework enables to assign physical IRQs to guests. >>> >>> 1) user-side uses KVM_IRQFD VM ioctl to pass KVM a kvm_irqfd struct that >>> associates a VM, an eventfd, an IRQ number (aka. the GSI). When an actor >>> signals the eventfd (typically a VFIO platform driver), the irqfd subsystem >>> injects the specified IRQ into the VM (the "GSI" takes the semantic of a >>> virtual IRQ for that guest). >> >> Just so I can understand how this works: Who EOIs (handles) the physical >> interrupt? If it is the VFIO driver, then I don't see how you prevent >> the interrupt from firing again immediately (unless this is an edge >> interrupt?). >> >>> 2) the other use case is user-side does 1) and uses KVM_SET_GSI_ROUTING >>> VM ioctl to create an association between a VM, a physical IRQ (aka GSI) and >>> a virtual IRQ (aka irchip.pin). This creates a so-called GSI routing entry. >>> When someone triggers the eventfd, irqfd handles it but uses the specified >>> routing and eventually injects irqchip.pin virtual IRQ into the guest. In >>> that >>> context the GSI takes the semantic of a physical IRQ while the irqchip.pin >>> takes the semantic of a virtual IRQ. >>> >>> in 1) routing is used by irqfd but an identity routing is created by default >>> making the gsi = irqchip.pin. Note on ARM there is a single interrupt >>> controller kind, the GIC. >>> >>> GSI routing mostly is implemented in generic irqchip.c. >>> The tiny ARM specific part is directly implemented in the virtual interrupt >>> controller (vgic.c) as it is done for powerpc for instance. This option was >>> prefered compared to implementing other #ifdef in irq_comm.c (x86 and ia64). >>> Hence irq_comm.c is not used at all. >>> >>> Routing currently is not used for anything else than irqfd IRQ injection. >>> Only >>> SPI can be injected. This means the vgic is not totally hidden behind the >>> irqchip. There are separate discussions on PPI/SGI routing. >>> >>> Only level sensitive IRQs are supported (with a registered resampler). As a >>> reminder the resampler is a second eventfd called by irqfd framework when >>> the >>> virtual IRQ is completed by the guest. This eventfd is supposed to be >>> handled >>> on user-side >>> >>> MSI routing is not supported yet. >>> >>> This work was tested with Calxeda Midway xgmac main interrupt (with and >>> without >>> explicit user routing) with qemu-system-arm and QEMU VFIO platform device. >>> >>> changes v1 -> v2: >>> 2 fixes: >>> - v1 assumed gsi/irqchip.pin was already incremented by >>> VGIC_NR_PRIVATE_IRQS. >>> This is now vgic_set_assigned_irq that increments it before injection. >>> - v2 now handles the case where a pending assigned irq is cleared through >>> MMIO access. The irq is properly acked allowing the resamplefd handler >>> to possibly unmask the physical IRQ. >>> >>> Signed-off-by: Eric Auger >>> >>> Conflicts: >>> Documentation/virtual/kvm/api.txt >>> arch/arm/kvm/Kconfig >>> >>> Conflicts: >>> Documentation/virtual/kvm/api.txt >>> --- >>> Documentation/virtual/kvm/api.txt | 4 +- >>> arch/arm/include/uapi/asm/kvm.h | 8 +++ >>> arch/arm/kvm/Kconfig | 2 + >>> arch/arm/kvm/Makefile | 1 + >>> arch/arm/kvm/irq.h| 25 +++ >>> virt/kvm/arm/vgic.c | 141 >>> -- >>> 6 files changed, 174 insertions(+), 7 deletions(-) >>> create mode 100644 arch/arm/kvm/irq.h >>> >>> diff --git a/Documentation/virtual/kvm/api.txt >>> b/Documentation/virtual/kvm/api.txt >>> index b4f5365..b376334 100644 >>> --- a/Documentation/virtual/kvm/api.txt >>> +++ b/Documentation/virtual/kvm/api.txt >>> @@ -1339,7 +1339,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or >>> guest IRQ is allowed. >>> 4.52 KVM_SET_GSI_ROUTING >>
Re: [RFC PATCH 4/6] arm/arm64: KVM: vgic: Improve handling of GICD_I{CS}PENDRn
On 06/14/2014 10:51 PM, Christoffer Dall wrote:
> The handling of writes to the GICD_ISPENDRn and GICD_ICPENDRn is
> currently not handled correctly for level-triggered interrupts.
Hi Christoffer,
Thanks for this patch serie. I can confirm it fixes my QEMU/VFIO issue
where all IRQs were pending cleared at guest OS boot while IRQ wires
were set. Now those IRQs are left pending which is compliant with the
GIC spec. You will find few comments/questions below.
Best Regards
Eric
> spec states that for level-triggered interrupts, writes to the
> GICD_ISPENDRn activates the output of a flip-flop which is in turn or'ed
> with the actual input interrupt signal. Correspondingly, writes to
> GICD_ICPENDRn simply deactives the output of that flip-flop, but does
deactivates
> not (of course) affect the external input signal. Reads from GICC_IAR
> will also deactivate the flip-flop output.
>
> This requires us to track the state of the level-input separately from
> the state in the flip-flop. Introduce two new variables on the
> distributor struct to track these two exact states. Astute readers
> may notice that this is introducing more state than required (because an
> OR of the two states give you the pending state), but the remainding
remaining
> vgic code uses the pending bitmap for optimized operations to figure
> out, at the end of the day, if an interrupt is pending or not on the
> distributor side. Changing all the to consider the two state variables
sentence
> did not look pretty.
>
> Signed-off-by: Christoffer Dall
> ---
> include/kvm/arm_vgic.h | 16 ++-
> virt/kvm/arm/vgic.c| 118
> -
> 2 files changed, 122 insertions(+), 12 deletions(-)
>
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index 10fa64b..f678d5c 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -86,9 +86,23 @@ struct vgic_dist {
> /* Interrupt enabled (one bit per IRQ) */
> struct vgic_bitmap irq_enabled;
>
> - /* Interrupt state is pending on the distributor */
> + /* Level-triggered interrupt external input is asserted */
> + struct vgic_bitmap irq_level;
was a bit confused by the name. I now understand this is the wire
(interrupt signal to GIC), relevant in case of level-sensitive IRQ.
~ wire_level.
> +
> + /*
> + * Interrupt state is pending on the distributor
> + */
> struct vgic_bitmap irq_pending;
~ status_includes_pending
>
> + /*
> + * Tracks writes to GICD_ISPENDRn and GICD_ICPENDRn for level-triggered
> + * interrupts. Essentially holds the state of the flip-flop in
> + * Figure 4-10 on page 4-101 in ARM IHI 0048B.b.
> + * Once set, it is only cleared for level-triggered interrupts on
> + * guest ACKs (when we queue it) or writes to GICD_ICPENDRn.
> + */
> + struct vgic_bitmap irq_soft_pend;
> +
> /* Level-triggered interrupt queued on VCPU interface */
> struct vgic_bitmap irq_queued;
>
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 87c977c..0b41875 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -67,6 +67,11 @@
> * - When the interrupt is EOIed, the maintenance interrupt fires,
> * and clears the corresponding bit in irq_queued. This allow the
> * interrupt line to be sampled again.
> + * - Note that level-triggered interrupts can also be set to pending from
> + * writes to GICD_ISPENDRn and lowering the external input line does not
> + * cause the interrupt to become inactive in such a situation.
> + * Conversely, writes to GICD_ICPENDRn do not cause the interrupt to become
> + * inactive as long as the external input line is held high.
> */
>
> #define VGIC_ADDR_UNDEF (-1)
> @@ -200,6 +205,41 @@ static void vgic_irq_clear_queued(struct kvm_vcpu *vcpu,
> int irq)
> vgic_bitmap_set_irq_val(&dist->irq_queued, vcpu->vcpu_id, irq, 0);
> }
>
> +static int vgic_dist_irq_get_level(struct kvm_vcpu *vcpu, int irq)
> +{
> + struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> + return vgic_bitmap_get_irq_val(&dist->irq_level, vcpu->vcpu_id, irq);
> +}
> +
> +static void vgic_dist_irq_set_level(struct kvm_vcpu *vcpu, int irq)
> +{
> + struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> + vgic_bitmap_set_irq_val(&dist->irq_level, vcpu->vcpu_id, irq, 1);
> +}
> +
> +static void vgic_dist_irq_clear_level(struct kvm_vcpu *vcpu, int irq)
> +{
> + struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> + vgic_bitmap_set_irq_val(&dist->irq_level, vcpu->vcpu_id, irq, 0);
> +}
> +
> +static int vgic_dist_irq_soft_pend(struct kvm_vcpu *vcpu, int irq)
> +{
> + struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> + return vgic_bitmap_get_irq_val(&dist->irq_soft_pend, vcpu->vcpu_id,
> irq);
> +}
> +
> +static void vgic_dist_irq_clear_soft_pend(struct kvm_vcpu *vcpu, int irq)
> +{
> + struct vgic_dist
Re: [RFC PATCH 1/6] arm/arm64: KVM: Rename irq_state to irq_pending
On 06/14/2014 10:51 PM, Christoffer Dall wrote:
> The irq_state field on the distributor struct is ambiguous in its
> meaning; the comment says it's the level of the input put, but that
> doesn't make much sense for edge-triggered interrupts. The code
> actually uses this state variable to check if the interrupt is in the
> pending state on the distributor so clarify the comment and rename the
> actual variable and accessor methods.
>
> Signed-off-by: Christoffer Dall
> ---
> include/kvm/arm_vgic.h | 4 ++--
> virt/kvm/arm/vgic.c| 52
> +-
> 2 files changed, 28 insertions(+), 28 deletions(-)
>
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index f27000f..a5d55d5 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -86,8 +86,8 @@ struct vgic_dist {
> /* Interrupt enabled (one bit per IRQ) */
> struct vgic_bitmap irq_enabled;
>
> - /* Interrupt 'pin' level */
> - struct vgic_bitmap irq_state;
> + /* Interrupt state is pending on the distributor */
> + struct vgic_bitmap irq_pending;
>
> /* Level-triggered interrupt in progress */
> struct vgic_bitmap irq_active;
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 47b2983..aba960a 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -37,7 +37,7 @@
> *
> * - At any time, the dist->irq_pending_on_cpu is the oracle that knows if
> * something is pending
> - * - VGIC pending interrupts are stored on the vgic.irq_state vgic
> + * - VGIC pending interrupts are stored on the vgic.irq_pending vgic
> * bitmap (this bitmap is updated by both user land ioctls and guest
> * mmio ops, and other in-kernel peripherals such as the
> * arch. timers) and indicate the 'wire' state.
Hi Christoffer,
Shouldn't we say that vgic.irq_pending is a combination of wire state
and soft pending set action(ISPENDR), corresponding to GIC archi spec
"status_includes_pending" - at least for level sensitive IRQS - ?
Best Regards
Eric
> @@ -45,8 +45,8 @@
> * recalculated
> * - To calculate the oracle, we need info for each cpu from
> * compute_pending_for_cpu, which considers:
> - * - PPI: dist->irq_state & dist->irq_enable
> - * - SPI: dist->irq_state & dist->irq_enable & dist->irq_spi_target
> + * - PPI: dist->irq_pending & dist->irq_enable
> + * - SPI: dist->irq_pending & dist->irq_enable & dist->irq_spi_target
> * - irq_spi_target is a 'formatted' version of the GICD_ICFGR
> * registers, stored on each vcpu. We only keep one bit of
> * information per interrupt, making sure that only one vcpu can
> @@ -204,21 +204,21 @@ static int vgic_dist_irq_is_pending(struct kvm_vcpu
> *vcpu, int irq)
> {
> struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>
> - return vgic_bitmap_get_irq_val(&dist->irq_state, vcpu->vcpu_id, irq);
> + return vgic_bitmap_get_irq_val(&dist->irq_pending, vcpu->vcpu_id, irq);
> }
>
> -static void vgic_dist_irq_set(struct kvm_vcpu *vcpu, int irq)
> +static void vgic_dist_irq_set_pending(struct kvm_vcpu *vcpu, int irq)
> {
> struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>
> - vgic_bitmap_set_irq_val(&dist->irq_state, vcpu->vcpu_id, irq, 1);
> + vgic_bitmap_set_irq_val(&dist->irq_pending, vcpu->vcpu_id, irq, 1);
> }
>
> -static void vgic_dist_irq_clear(struct kvm_vcpu *vcpu, int irq)
> +static void vgic_dist_irq_clear_pending(struct kvm_vcpu *vcpu, int irq)
> {
> struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>
> - vgic_bitmap_set_irq_val(&dist->irq_state, vcpu->vcpu_id, irq, 0);
> + vgic_bitmap_set_irq_val(&dist->irq_pending, vcpu->vcpu_id, irq, 0);
> }
>
> static void vgic_cpu_irq_set(struct kvm_vcpu *vcpu, int irq)
> @@ -392,7 +392,7 @@ static bool handle_mmio_set_pending_reg(struct kvm_vcpu
> *vcpu,
> struct kvm_exit_mmio *mmio,
> phys_addr_t offset)
> {
> - u32 *reg = vgic_bitmap_get_reg(&vcpu->kvm->arch.vgic.irq_state,
> + u32 *reg = vgic_bitmap_get_reg(&vcpu->kvm->arch.vgic.irq_pending,
> vcpu->vcpu_id, offset);
> vgic_reg_access(mmio, reg, offset,
> ACCESS_READ_VALUE | ACCESS_WRITE_SETBIT);
> @@ -408,7 +408,7 @@ static bool handle_mmio_clear_pending_reg(struct kvm_vcpu
> *vcpu,
> struct kvm_exit_mmio *mmio,
> phys_addr_t offset)
> {
> - u32 *reg = vgic_bitmap_get_reg(&vcpu->kvm->arch.vgic.irq_state,
> + u32 *reg = vgic_bitmap_get_reg(&vcpu->kvm->arch.vgic.irq_pending,
> vcpu->vcpu_id, offset);
> vgic_reg_access(mmio, reg, offset,
> ACCESS_READ_VALUE | ACCESS_WRITE_CLEARBIT);
> @@ -650,7 +650,7 @@ static void vgic_unqueue_irqs(struct kvm_vcpu *vcpu)
>
Re: [RFC PATCH 6/6] arm/arm64: KVM: vgic: Clarify and correct vgic documentation
On 06/14/2014 10:51 PM, Christoffer Dall wrote: > The VGIC virtual distributor implementation documentation was written a > very long time ago, before the true nature of the beast had been > partially absorbed into my bloodstream. I think this amalgamates the > two evil beings (myself and the code) a little more. > > Plus, it fixes an actual bug. ICFRn, pfff. > > Signed-off-by: Christoffer Dall > --- > virt/kvm/arm/vgic.c | 13 +++-- > 1 file changed, 7 insertions(+), 6 deletions(-) > > diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c > index 1f91b3b..cc776af 100644 > --- a/virt/kvm/arm/vgic.c > +++ b/virt/kvm/arm/vgic.c > @@ -36,21 +36,22 @@ > * How the whole thing works (courtesy of Christoffer Dall): > * > * - At any time, the dist->irq_pending_on_cpu is the oracle that knows if > - * something is pending > - * - VGIC pending interrupts are stored on the vgic.irq_pending vgic > - * bitmap (this bitmap is updated by both user land ioctls and guest > - * mmio ops, and other in-kernel peripherals such as the > - * arch. timers) and indicate the 'wire' state. > + * something is pending on the CPU interface. > + * - Interrupts that are pending on the distributor are stored on the > + * vgic.irq_pending vgic bitmap (this bitmap is updated by both user land > + * ioctls and guest mmio ops, and other in-kernel peripherals such as the > + * arch. timers). ok forget my previous comment related to wire state;-) > * - Every time the bitmap changes, the irq_pending_on_cpu oracle is > * recalculated > * - To calculate the oracle, we need info for each cpu from > * compute_pending_for_cpu, which considers: > * - PPI: dist->irq_pending & dist->irq_enable > * - SPI: dist->irq_pending & dist->irq_enable & dist->irq_spi_target > - * - irq_spi_target is a 'formatted' version of the GICD_ICFGR > + * - irq_spi_target is a 'formatted' version of the GICD_ITARGETSRn > * registers, stored on each vcpu. We only keep one bit of > * information per interrupt, making sure that only one vcpu can > * accept the interrupt. > + * - If any of the above state changes, we must recalculate the oracle. > * - The same is true when injecting an interrupt, except that we only > * consider a single interrupt at a time. The irq_spi_cpu array > * contains the target CPU for each SPI. > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Coupling between KVM_IRQFD and KVM_SET_GSI_ROUTING?
Hi, If I am not wrong I did not get any reply to this question. Can anyone share his/her experience/knowledge on this gsi routing/irqfd use case? Thank you in advance Best Regards Eric On 06/17/2014 01:39 PM, Eric Auger wrote: > Hello, > > I have a question related to KVM_IRQFD and KVM_SET_GSI_ROUTING ioctl > relationship. > > When reading the KVM API documentation I do not understand there is any > dependency between KVM_IRQFD and KVM_SET_GSI_ROUTING. According to the > text it seems only the gsi field is used and interpreted as the irqchip pin. > > However irqchip.c kvm_set_irq code relies on an existing and not dummy > routing table. > > My question is: does anyone agree on the fact the user-side must set a > consistent routing table using KVM_SET_GSI_ROUTING before using > KVM_IRQFD? The other alternative would have been to build a default > identity GSI routing table in the kernel (gsi = irqchip.pin). > > In the positive, shouldn't we clarify the KVM API documentation? > > Thank you in advance > > Best Regards > > Eric > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] ARM: KVM: add irqfd and irq routing support
On 06/19/2014 04:13 PM, Will Deacon wrote:
> Hi all,
>
> I'm currently adding VFIO support for kvmtool, so I'm interested in this
> patch series (although actually from a PCI perspective).
>
> Eric: can you CC me on future versions of this series please? Once things
> start to stabilise, I can help with testing.
Hi Will,
sure I will CC you.
>
> On Thu, Jun 05, 2014 at 03:39:50PM +0100, Christoffer Dall wrote:
>> On Thu, Jun 05, 2014 at 03:15:15PM +0200, Eric Auger wrote:
>>> On 06/05/2014 12:28 PM, Christoffer Dall wrote:
>>>> On Mon, Jun 02, 2014 at 09:29:56AM +0200, Eric Auger wrote:
>>>>> + kvm_debug("Inject irqchip routed vIRQ %d\n",
>>>>> + e->irqchip.pin);
>>>>> + kvm_vgic_inject_irq(kvm, 0, spi, level);
>>>>> + /*
>>>>> + * toggling down vIRQ wire is directly handled in
>>>>> + * process_maintenance for this reason:
>>>>> + * irqfd_resampler_ack is called in
>>>>> + * process_maintenance which holds the dist lock.
>>>>> + * irqfd_resampler_ack calls kvm_set_irq
>>>>> + * which ends_up calling kvm_vgic_inject_irq.
>>>>> + * This later attempts to take the lock -> deadlock!
>>>>> + */
>>>>
>>>> Not sure I understand this comment. What are we trying to achieve, are
>>>> we using some sort of a workaround to avoid a deadlock?
>>>
>>> What I wanted to point out here is I would have prefered to handle both
>>> levels 0 and 1 in a symetrical manner. irqfd_resampler_ack (in eventfd)
>>> is calling kvm_set_irq with level 0. This would be the prefered way to
>>> toggle down the SPI at GIC input instead of doing this in
>>> process_maintenance in a dirty manner. However this does work because
>>> irqfd_resampler_ack is called in process_maintenance (the place where
>>> the EOI is analyzed). process_maintenance holds the dist lock and would
>>> eventually call kvm_vgic_inject_irq which also attempts to take the lock.
>>>
>>
>> I'm afraid that's too much of a hack. There's an external mechanism to
>> set an interrupt line to active (level=1) or inactive (level=0) and we
>> must support both.
>>
>> The fact that vgic_process_maintenance() can set the interrupt line to
>> inactive is just something we exploit to properly handle level-triggered
>> interrupts, but the main API to the VGIC must absolutely be supported.
>>
>> Am I completely wrong here?
>>
>> The locking issue can be solved by splitting up the locking into a finer
>> granularity as needed or deferring the call to irqfd_resampler_ack()
>> until after unlocking the distributor lock in kvm_vgic_sync_hwstate().
>
> Why can't we do what PowerPC does for mpic and x86 does for IOAPIC and
> simply drop the distributor lock across the call to kvm_notify_acked_irq?
Yes, I am about to release a new version for this RFC that uses a finer
granularity for the dist lock, as you and Christoffer suggested.
>
> Given that I think the eventfd callbacks can block, holding a spinlock isn't
> safe anyway, regardless of the vgic re-entrancy issue.
yes you're fully right.
Best Regards
Eric
>
> Will
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v3] ARM: KVM: add irqfd and irq routing support
This patch enables irqfd and irq routing on ARM. It turns CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQ_ROUTING on. irqfd framework enables to inject a virtual IRQ into a guest upon an eventfd trigger. 1) user-side first needs to setup a GSI routing table using KVM_SET_GSI_ROUTING ioctl. A routing entry defines an association between an IRQ (aka GSI) and an irqchip pin. On ARM there is a single irqchip, ie. the GIC. On ARM, natural choice is to set gsi = irqchip.pin. 2) user-side uses KVM_IRQFD VM ioctl to provide KVM with a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number (aka. the GSI). When an actor signals the eventfd (typically a VFIO platform driver), the irqfd subsystem injects a virtual IRQ correponding to the irqchip pin associated to that GSI. irqchip.pin is computed from previous routing table. On ARM it is assumed to by an SPI only. This RFC applies on top of https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git branch irqfd_integ_v3 Signed-off-by: Eric Auger --- GSI routing mostly is implemented in generic irqchip.c. The tiny ARM specific part is directly implemented in the virtual interrupt controller (vgic.c) as it is done for powerpc for instance. This option was prefered compared to implementing other #ifdef in irq_comm.c (x86 and ia64). Hence irq_comm.c is not used at all. MSI routing is not supported yet. Edge sensitive IRQ injection was not tested but should be OK (KVM_USERSPACE_IRQ_SOURCE_ID path). This work was tested with Calxeda Midway xgmac main interrupt with qemu-system-arm and QEMU VFIO platform device. Known issues: - static allocation of chip[][] in irqchip.c forces to statically dimension the number of IRQS supported by the VGIC. KVM_IRQCHIP_NUM_PINS still currently is set to VGIC_NR_IRQS, which may become VGIC_NR_IRQS_LEGACY with the advent of: http://www.spinics.net/lists/arm-kernel/msg277415.html - if for some reason the IRQ is never EOI'ed, the notifier never is called. If its job typically consists in unmasking the physical IRQ as it is for VFIO, the IRQ might stay masked. v3: - correct misc style issues - remove notifier call in clear pending MMIO write, now fixed by Christoffer VGIC clear pending correction: https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html - remove allocation of identity routing table. It is assumed to be user-side's job to set it. - vgic_set_assigned_irq now handles both levels in a symetrical way. dist lock issue fixed by defining finer lock regions in kvm_vgic_sync_hwstate() - IRQFD implementation better documented in kvm/api.txt - KVM_IRQCHIP_NUM_PINS set to VGIC_NR_IRQS_LEGACY as temporary solution - check ue->u.irqchip.irqchip v2: 2 fixes: - v1 assumed gsi/irqchip.pin was already incremented by VGIC_NR_PRIVATE_IRQS. This is now vgic_set_assigned_irq that increments it before injection. - v2 now handles the case where a pending assigned irq is cleared through MMIO access. The irq is properly acked allowing the resamplefd handler to possibly unmask the physical IRQ. --- Documentation/virtual/kvm/api.txt | 12 - arch/arm/include/uapi/asm/kvm.h | 9 arch/arm/kvm/Kconfig | 2 + arch/arm/kvm/Makefile | 1 + arch/arm/kvm/irq.h| 25 + virt/kvm/arm/vgic.c | 108 -- 6 files changed, 151 insertions(+), 6 deletions(-) create mode 100644 arch/arm/kvm/irq.h diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt index b4f5365..326e382 100644 --- a/Documentation/virtual/kvm/api.txt +++ b/Documentation/virtual/kvm/api.txt @@ -1339,7 +1339,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed. 4.52 KVM_SET_GSI_ROUTING Capability: KVM_CAP_IRQ_ROUTING -Architectures: x86 ia64 s390 +Architectures: x86 ia64 s390 arm Type: vm ioctl Parameters: struct kvm_irq_routing (in) Returns: 0 on success, -1 on error @@ -2126,7 +2126,7 @@ into the hash PTE second double word). 4.75 KVM_IRQFD Capability: KVM_CAP_IRQFD -Architectures: x86 s390 +Architectures: x86 s390 arm Type: vm ioctl Parameters: struct kvm_irqfd (in) Returns: 0 on success, -1 on error @@ -2152,6 +2152,14 @@ Note that closing the resamplefd is not sufficient to disable the irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment and need not be specified with KVM_IRQFD_FLAG_DEASSIGN. +On ARM/arm64 the virtual IRQ injection mandates the existence of a +routing table, set through KVM_SET_GSI_ROUTING. this latter contains +entries which associate a gsi with an irqchip pin. The injected virtual +IRQ actually corresponds to the irqchip.pin. It is up to the user to +define gsi = irqchip.pin or not. On ARM the single irqchip is the GIC. +Then irqchip.pin is interpreted as the system shared peripheral +interrupt number (SPI). Associated GIC in
Re: Coupling between KVM_IRQFD and KVM_SET_GSI_ROUTING?
On 06/24/2014 12:40 PM, Alexander Graf wrote: > > On 24.06.14 11:47, Paul Mackerras wrote: >> On Mon, Jun 23, 2014 at 06:47:51PM +0200, Alexander Graf wrote: >>> On 17.06.14 13:39, Eric Auger wrote: >>>> Hello, >>>> >>>> I have a question related to KVM_IRQFD and KVM_SET_GSI_ROUTING ioctl >>>> relationship. >>>> >>>> When reading the KVM API documentation I do not understand there is any >>>> dependency between KVM_IRQFD and KVM_SET_GSI_ROUTING. According to the >>>> text it seems only the gsi field is used and interpreted as the >>>> irqchip pin. >>>> >>>> However irqchip.c kvm_set_irq code relies on an existing and not dummy >>>> routing table. >>>> >>>> My question is: does anyone agree on the fact the user-side must set a >>>> consistent routing table using KVM_SET_GSI_ROUTING before using >>>> KVM_IRQFD? The other alternative would have been to build a default >>>> identity GSI routing table in the kernel (gsi = irqchip.pin). >>> I untangled irqfd support from the x86 ioapic emulation a while back. >>> When I >>> looked at it, I didn't see any easy way to split it out from the routing >>> too, so I kept that dependency in. Hi Alex, Paul, thanks for your replies. hence don't you think the KVM API doc deserves to be clarified then? >>> >>> If you look at the code, you will see that the irq routing entry is >>> used as >>> token for an irqfd. So every irqfd only knows its routing table entry, >>> nothing else. >> As far as I can see, the routing table entry is used for only two >> things: to tell whether the interrupt is an MSI or LSI, and to pass to >> kvm_set_msi. Indeed I noticed irq_entry in _irqfd struct only is set for MSI routing. For other IRQS - I guess what you call LSI - , routine table map[] is used to retrieve the irchip.pin(s) associated to that gsi and call set(). >> >> One way to tackle the problem would be to abstract out the routing >> table lookup into a function in irqchip.c, rather than having it >> open-coded in irqfd_update. Then, if you don't want to have the >> routing table you write an alternative function that creates a >> routing entry from scratch. It would need information from the >> low-level irq chip code as to which interrupts are LSIs and which are >> MSIs. It also ties you down to having only one kind of interrupt controller. > > You could also create it along with the irqfd state struct. So instead of > > kzalloc(sizeof(*irqfd), GFP_KERNEL); > > we could do > > kzalloc(sizeof(*irqfd) + sizeof(struct kvm_kernel_irq_routing_entry), > GFP_KERNEL); Wouldn't it make sense, in kvm_irqfd_assign, to test whether there is a routing entry associated to that gsi. In the negative, create a default one where irqchip.pin = gsi or something architecture specific?. That way would not need a complete & big routing table to be set by user-side? > > and point the routing entry to the inline irqfd one. We'd lose the > ability to change any routings with that construct, but I think that's > ok for irq controllers that are not configurable. > > Alternatively, we could also have a single routing entry that is a > wildcard match, ranging from say LSI [x...y]. The irqfd structure would > then need to carry a local payload to define an offset inside that > wildcard routing entry. Also whatever the number of IRQs assigned, we have this big chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] allocated. Also on ARM we have some difficulties to define KVM_IRQCHIP_NUM_PINS which becomes quite dynamic. Best Regards Eric > > > Alex > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Coupling between KVM_IRQFD and KVM_SET_GSI_ROUTING?
On 06/24/2014 05:47 PM, Alexander Graf wrote: > > On 24.06.14 17:05, Eric Auger wrote: >> On 06/24/2014 12:40 PM, Alexander Graf wrote: >>> On 24.06.14 11:47, Paul Mackerras wrote: >>>> On Mon, Jun 23, 2014 at 06:47:51PM +0200, Alexander Graf wrote: >>>>> On 17.06.14 13:39, Eric Auger wrote: >> >>>>>> Hello, >>>>>> >>>>>> I have a question related to KVM_IRQFD and KVM_SET_GSI_ROUTING ioctl >>>>>> relationship. >>>>>> >>>>>> When reading the KVM API documentation I do not understand there >>>>>> is any >>>>>> dependency between KVM_IRQFD and KVM_SET_GSI_ROUTING. According to >>>>>> the >>>>>> text it seems only the gsi field is used and interpreted as the >>>>>> irqchip pin. >>>>>> >>>>>> However irqchip.c kvm_set_irq code relies on an existing and not >>>>>> dummy >>>>>> routing table. >>>>>> >>>>>> My question is: does anyone agree on the fact the user-side must >>>>>> set a >>>>>> consistent routing table using KVM_SET_GSI_ROUTING before using >>>>>> KVM_IRQFD? The other alternative would have been to build a default >>>>>> identity GSI routing table in the kernel (gsi = irqchip.pin). >>>>> I untangled irqfd support from the x86 ioapic emulation a while back. >>>>> When I >>>>> looked at it, I didn't see any easy way to split it out from the >>>>> routing >>>>> too, so I kept that dependency in. >> Hi Alex, Paul, >> >> thanks for your replies. hence don't you think the KVM API doc deserves >> to be clarified then? >>>>> If you look at the code, you will see that the irq routing entry is >>>>> used as >>>>> token for an irqfd. So every irqfd only knows its routing table entry, >>>>> nothing else. >>>> As far as I can see, the routing table entry is used for only two >>>> things: to tell whether the interrupt is an MSI or LSI, and to pass to >>>> kvm_set_msi. >> >> Indeed I noticed irq_entry in _irqfd struct only is set for MSI routing. >> For other IRQS - I guess what you call LSI - , routine table map[] is >> used to retrieve the irchip.pin(s) associated to that gsi and call set(). >>>> One way to tackle the problem would be to abstract out the routing >>>> table lookup into a function in irqchip.c, rather than having it >>>> open-coded in irqfd_update. Then, if you don't want to have the >>>> routing table you write an alternative function that creates a >>>> routing entry from scratch. It would need information from the >>>> low-level irq chip code as to which interrupts are LSIs and which are >>>> MSIs. It also ties you down to having only one kind of interrupt >> controller. >>> You could also create it along with the irqfd state struct. So >>> instead of >>> >>>kzalloc(sizeof(*irqfd), GFP_KERNEL); >>> >>> we could do >>> >>>kzalloc(sizeof(*irqfd) + sizeof(struct kvm_kernel_irq_routing_entry), >>> GFP_KERNEL); >> >> Wouldn't it make sense, in kvm_irqfd_assign, to test whether there is a >> routing entry associated to that gsi. In the negative, create a default >> one where irqchip.pin = gsi or something architecture specific?. > > "no entry" really should be "no entry". I think if we want some other > mechanism, we should make it be explicit. Yep I agree. But currently the usage of routing mechanism is not explicit neither in 4.75 KVM_IRQFD and that was causing my doubts;-) On top of that the doc says: "kvm_irqfd.gsi specifies the irqchip pin toggled by this event" but if there is no routing table set before by user-side, the injection will fail. > >> >> That way would not need a complete & big routing table to be set by >> user-side? >> >>> and point the routing entry to the inline irqfd one. We'd lose the >>> ability to change any routings with that construct, but I think that's >>> ok for irq controllers that are not configurable. >>> >>> Alternatively, we could also have a single routing entry that is a >>> wildcard match, ranging from say LSI [x...y]. The irqfd structure would >>> then need to carry a local payload to define an offset inside that >>> wildcard routing entry. >> Also whatever the number of IRQs assigned, we have this big >> chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS] allocated. Also on ARM we >> have some difficulties to define KVM_IRQCHIP_NUM_PINS which becomes >> quite dynamic. > > Right, if you go with the ranged routing entries you should be able to > get by with a very small number of NUM_PINS, no? Yes that's correct. but the effort to integrate such feature in the code need to be further assessed. Eric > > > Alex > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/6] IRQFD without IRQ routing, enabled for XICS
Hi Paul, I also confirm your patch works fine on my ARM test environment. I was able to run - with irqchip without regression - without irqchip (ie removing routing totally), just implementing identity kvm_irq_map_gsi and kvm_irq_map_chip_pin and proper kvm_set_irq. The overall integration becomes much simpler :-) Many thanks Best Regards Eric On 07/14/2014 04:18 PM, Cornelia Huck wrote: > On Mon, 30 Jun 2014 20:51:08 +1000 > Paul Mackerras wrote: > >> I would like to see this go into 3.17. > > FWIW: I've given this a whirl on s390 (with a dataplane disk), and > everything seems to work as before. > > The only thing which is I think worth mentioning is that embedding the > routing entry into the irqfd struct will grow it a bit, which might be > noticable on large installations with hundreds of devices. OTOH, the > routing entry isn't too large, so I don't think it will become a > problem. > >> >> arch/ia64/kvm/Kconfig| 1 + >> arch/powerpc/kvm/Kconfig | 3 + >> arch/powerpc/kvm/book3s_hv_rm_xics.c | 5 ++ >> arch/powerpc/kvm/book3s_xics.c | 55 +++--- >> arch/powerpc/kvm/book3s_xics.h | 2 + >> arch/powerpc/kvm/mpic.c | 4 +- >> arch/s390/kvm/Kconfig| 1 + >> arch/s390/kvm/interrupt.c| 3 +- >> arch/x86/kvm/Kconfig | 1 + >> include/linux/kvm_host.h | 43 --- >> virt/kvm/Kconfig | 3 + >> virt/kvm/eventfd.c | 134 >> ++- >> virt/kvm/irq_comm.c | 24 +++ >> virt/kvm/irqchip.c | 98 ++--- >> virt/kvm/kvm_main.c | 2 +- >> 15 files changed, 227 insertions(+), 152 deletions(-) > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC PATCH] ARM: KVM: add irqfd support
This patch enables irqfd on ARM.
irqfd framework enables to inject a virtual IRQ into a guest upon an
eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number
(aka. the gsi). When an actor signals the eventfd (typically a VFIO
platform driver), the kvm irqfd subsystem injects the provided virtual
IRQ into the guest.
The gsi must correspond to a shared peripheral interrupt (SPI), ie the
GIC interrupt ID is gsi+32.
CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD are turned on.
No IRQ routing table is used thanks to Paul Mackerras' patch serie:
"IRQFD without IRQ routing, enabled for XICS"
(https://www.mail-archive.com/kvm@vger.kernel.org/msg104478.html)
Signed-off-by: Eric Auger
---
This patch would deprecate the previous patch featuring GSI routing
(https://patches.linaro.org/32261/)
irqchip.c and irq_comm.c are not used at all.
This RFC applies on top of Christoffer Dall's serie
arm/arm64: KVM: Various VGIC cleanups and improvements
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
branch irqfd_integ_v4
This work was tested with Calxeda Midway xgmac main interrupt with
qemu-system-arm and QEMU VFIO platform device.
---
Documentation/virtual/kvm/api.txt | 5 +++-
arch/arm/include/uapi/asm/kvm.h | 3 +++
arch/arm/kvm/Kconfig | 3 ++-
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/irq.h| 25 ++
virt/kvm/arm/vgic.c | 54 ---
6 files changed, 85 insertions(+), 7 deletions(-)
create mode 100644 arch/arm/kvm/irq.h
diff --git a/Documentation/virtual/kvm/api.txt
b/Documentation/virtual/kvm/api.txt
index 0fe3649..04310d9 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2132,7 +2132,7 @@ into the hash PTE second double word).
4.75 KVM_IRQFD
Capability: KVM_CAP_IRQFD
-Architectures: x86 s390
+Architectures: x86 s390 arm
Type: vm ioctl
Parameters: struct kvm_irqfd (in)
Returns: 0 on success, -1 on error
@@ -2158,6 +2158,9 @@ Note that closing the resamplefd is not sufficient to
disable the
irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
+On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
+This means the programmed GIC interrupt ID is gsi+32.
+
4.76 KVM_PPC_ALLOCATE_HTAB
Capability: KVM_CAP_PPC_ALLOC_HTAB
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index e6ebdd3..3034c66 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
/* Highest supported SPI, from VGIC_NR_IRQS */
#define KVM_ARM_IRQ_GIC_MAX127
+/* One single KVM irqchip, ie. the VGIC */
+#define KVM_NR_IRQCHIPS 1
+
/* PSCI interface */
#define KVM_PSCI_FN_BASE 0x95c1ba5e
#define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 4be5bb1..7800261 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -24,6 +24,7 @@ config KVM
select KVM_MMIO
select KVM_ARM_HOST
depends on ARM_VIRT_EXT && ARM_LPAE && !CPU_BIG_ENDIAN
+ select HAVE_KVM_EVENTFD
---help---
Support hosting virtualized guest machines. You will also
need to select one or more of the processor modules below.
@@ -55,7 +56,7 @@ config KVM_ARM_MAX_VCPUS
config KVM_ARM_VGIC
bool "KVM support for Virtual GIC"
depends on KVM_ARM_HOST && OF
- select HAVE_KVM_IRQCHIP
+ select HAVE_KVM_IRQFD
default y
---help---
Adds support for a hardware assisted, in-kernel GIC emulation.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 789bca9..2fa2f82 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
KVM := ../../../virt/kvm
-kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
+kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
diff --git a/arch/arm/kvm/irq.h b/arch/arm/kvm/irq.h
new file mode 100644
index 000..1275d91
--- /dev/null
+++ b/arch/arm/kvm/irq.h
@@ -0,0 +1,25 @@
+/*
+ * Copyright (C) 2014 Linaro Ltd.
+ * Authors: Eric Auger
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but W
Re: The status about vhost-net on kvm-arm?
On 08/12/2014 04:41 AM, Li Liu wrote: > Hi all, > > Is anyone there can tell the current status of vhost-net on kvm-arm? > > Half a year has passed from Isa Ansharullah asked this question: > http://www.spinics.net/lists/kvm-arm/msg08152.html > > I have found two patches which have provided the kvm-arm support of > eventfd and irqfd: > > 1) [RFC PATCH 0/4] ARM: KVM: Enable the ioeventfd capability of KVM on ARM > http://lists.gnu.org/archive/html/qemu-devel/2014-01/msg01770.html > > 2) [RFC,v3] ARM: KVM: add irqfd and irq routing support > https://patches.linaro.org/32261/ Hi Li, The patch below uses Paul Mackerras' work and removed usage of GSI routing table. It is a simpler alternative to 2) http://www.spinics.net/lists/kvm/msg106535.html > > And there's a rough patch for qemu to support eventfd from Ying-Shiuan Pan: > > [Qemu-devel] [PATCH 0/4] ioeventfd support for virtio-mmio > https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00715.html > > But there no any comments of this patch. And I can found nothing about qemu > to support irqfd. Do I lost the track? Actually I am using irqfd in QEMU VFIO Platform device https://lists.nongnu.org/archive/html/qemu-devel/2014-08/msg01455.html Best Regards Eric > > If nobody try to fix it. We have a plan to complete it about virtio-mmio > supporing irqfd and multiqueue. > > > > > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] ARM: KVM: add irqfd support
On 08/13/2014 04:55 PM, Christoffer Dall wrote:
> On Mon, Aug 04, 2014 at 02:08:22PM +0200, Eric Auger wrote:
>> This patch enables irqfd on ARM.
>>
>> irqfd framework enables to inject a virtual IRQ into a guest upon an
>> eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
>> a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number
>> (aka. the gsi). When an actor signals the eventfd (typically a VFIO
>> platform driver), the kvm irqfd subsystem injects the provided virtual
>> IRQ into the guest.
>>
>> The gsi must correspond to a shared peripheral interrupt (SPI), ie the
>> GIC interrupt ID is gsi+32.
>
> Why can't we support PPIs?
Hi Christoffer,
Well, in case we want to support PPI at irqfd level, we would need to
change the semantic of the GSI value and use the same as KVM_IRQ_LINE,
to specify the target vcpu. This is obviously feasible but this also
induces changes in currently generic user parts, vfio, vhost. is PPI
injection though irqfd a valid use case?
>
>>
>> CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD are turned on.
>
> This patch enables CONFIG_
>
OK
>>
>> No IRQ routing table is used thanks to Paul Mackerras' patch serie:
>> "IRQFD without IRQ routing, enabled for XICS"
>> (https://www.mail-archive.com/kvm@vger.kernel.org/msg104478.html)
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> This patch would deprecate the previous patch featuring GSI routing
>> (https://patches.linaro.org/32261/)
>>
>> irqchip.c and irq_comm.c are not used at all.
>>
>> This RFC applies on top of Christoffer Dall's serie
>> arm/arm64: KVM: Various VGIC cleanups and improvements
>> https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
>>
>> All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
>> branch irqfd_integ_v4
>>
>> This work was tested with Calxeda Midway xgmac main interrupt with
>> qemu-system-arm and QEMU VFIO platform device.
>> ---
>> Documentation/virtual/kvm/api.txt | 5 +++-
>> arch/arm/include/uapi/asm/kvm.h | 3 +++
>> arch/arm/kvm/Kconfig | 3 ++-
>> arch/arm/kvm/Makefile | 2 +-
>> arch/arm/kvm/irq.h| 25 ++
>> virt/kvm/arm/vgic.c | 54
>> ---
>> 6 files changed, 85 insertions(+), 7 deletions(-)
>> create mode 100644 arch/arm/kvm/irq.h
>>
>> diff --git a/Documentation/virtual/kvm/api.txt
>> b/Documentation/virtual/kvm/api.txt
>> index 0fe3649..04310d9 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2132,7 +2132,7 @@ into the hash PTE second double word).
>> 4.75 KVM_IRQFD
>>
>> Capability: KVM_CAP_IRQFD
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm
>> Type: vm ioctl
>> Parameters: struct kvm_irqfd (in)
>> Returns: 0 on success, -1 on error
>> @@ -2158,6 +2158,9 @@ Note that closing the resamplefd is not sufficient to
>> disable the
>> irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>> and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>
>> +On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
>> +This means the programmed GIC interrupt ID is gsi+32.
>> +
>> 4.76 KVM_PPC_ALLOCATE_HTAB
>>
>> Capability: KVM_CAP_PPC_ALLOC_HTAB
>> diff --git a/arch/arm/include/uapi/asm/kvm.h
>> b/arch/arm/include/uapi/asm/kvm.h
>> index e6ebdd3..3034c66 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
>> /* Highest supported SPI, from VGIC_NR_IRQS */
>> #define KVM_ARM_IRQ_GIC_MAX 127
>>
>> +/* One single KVM irqchip, ie. the VGIC */
>> +#define KVM_NR_IRQCHIPS 1
>> +
>> /* PSCI interface */
>> #define KVM_PSCI_FN_BASE0x95c1ba5e
>> #define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 4be5bb1..7800261 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -24,6 +24,7 @@ config KVM
>> select KVM_MMIO
>> select KVM_ARM_HOST
>> depends on ARM_VIRT_EXT && ARM_LPAE && !CPU_BIG_ENDIAN
>> +select HAVE_KVM_EVENTFD
>> ---help---
>>Support hosting virtualized guest machines. You will also
>>
Re: [RFC PATCH] ARM: KVM: add irqfd support
On 08/13/2014 04:55 PM, Christoffer Dall wrote:
> On Mon, Aug 04, 2014 at 02:08:22PM +0200, Eric Auger wrote:
>> This patch enables irqfd on ARM.
>>
>> irqfd framework enables to inject a virtual IRQ into a guest upon an
>> eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
>> a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number
>> (aka. the gsi). When an actor signals the eventfd (typically a VFIO
>> platform driver), the kvm irqfd subsystem injects the provided virtual
>> IRQ into the guest.
>>
>> The gsi must correspond to a shared peripheral interrupt (SPI), ie the
>> GIC interrupt ID is gsi+32.
>
> Why can't we support PPIs?
Hi Christoffer,
Well, in case we want to support PPI at irqfd level, we would need to
change the semantic of the GSI value and use the same as KVM_IRQ_LINE,
to specify the target vcpu. This is obviously feasible but this also
induces changes in currently generic user parts, vfio, vhost. is PPI
injection though irqfd a valid use case?
>
>>
>> CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD are turned on.
>
> This patch enables CONFIG_
>
OK
>>
>> No IRQ routing table is used thanks to Paul Mackerras' patch serie:
>> "IRQFD without IRQ routing, enabled for XICS"
>> (https://www.mail-archive.com/kvm@vger.kernel.org/msg104478.html)
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> This patch would deprecate the previous patch featuring GSI routing
>> (https://patches.linaro.org/32261/)
>>
>> irqchip.c and irq_comm.c are not used at all.
>>
>> This RFC applies on top of Christoffer Dall's serie
>> arm/arm64: KVM: Various VGIC cleanups and improvements
>> https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
>>
>> All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
>> branch irqfd_integ_v4
>>
>> This work was tested with Calxeda Midway xgmac main interrupt with
>> qemu-system-arm and QEMU VFIO platform device.
>> ---
>> Documentation/virtual/kvm/api.txt | 5 +++-
>> arch/arm/include/uapi/asm/kvm.h | 3 +++
>> arch/arm/kvm/Kconfig | 3 ++-
>> arch/arm/kvm/Makefile | 2 +-
>> arch/arm/kvm/irq.h| 25 ++
>> virt/kvm/arm/vgic.c | 54
>> ---
>> 6 files changed, 85 insertions(+), 7 deletions(-)
>> create mode 100644 arch/arm/kvm/irq.h
>>
>> diff --git a/Documentation/virtual/kvm/api.txt
>> b/Documentation/virtual/kvm/api.txt
>> index 0fe3649..04310d9 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2132,7 +2132,7 @@ into the hash PTE second double word).
>> 4.75 KVM_IRQFD
>>
>> Capability: KVM_CAP_IRQFD
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm
>> Type: vm ioctl
>> Parameters: struct kvm_irqfd (in)
>> Returns: 0 on success, -1 on error
>> @@ -2158,6 +2158,9 @@ Note that closing the resamplefd is not sufficient to
>> disable the
>> irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>> and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>
>> +On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
>> +This means the programmed GIC interrupt ID is gsi+32.
>> +
>> 4.76 KVM_PPC_ALLOCATE_HTAB
>>
>> Capability: KVM_CAP_PPC_ALLOC_HTAB
>> diff --git a/arch/arm/include/uapi/asm/kvm.h
>> b/arch/arm/include/uapi/asm/kvm.h
>> index e6ebdd3..3034c66 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
>> /* Highest supported SPI, from VGIC_NR_IRQS */
>> #define KVM_ARM_IRQ_GIC_MAX 127
>>
>> +/* One single KVM irqchip, ie. the VGIC */
>> +#define KVM_NR_IRQCHIPS 1
>> +
>> /* PSCI interface */
>> #define KVM_PSCI_FN_BASE0x95c1ba5e
>> #define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 4be5bb1..7800261 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -24,6 +24,7 @@ config KVM
>> select KVM_MMIO
>> select KVM_ARM_HOST
>> depends on ARM_VIRT_EXT && ARM_LPAE && !CPU_BIG_ENDIAN
>> +select HAVE_KVM_EVENTFD
>> ---help---
>>Support hosting virtualized guest machines. You will also
>>
[PATCH v2 0/2] irqfd support for ARM
This patch serie enables irqfd on ARM. irqfd framework enables to inject a virtual IRQ into a guest upon an eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number (aka. the gsi). When an actor signals the eventfd (typically a VFIO platform driver), the kvm irqfd subsystem injects the provided virtual IRQ into the guest. Resamplefd also is supported for level sensitive interrupts, ie. the user can provide another eventfd that is triggered when the completion of the virtual IRQ (gsi) is detected by the GIC. The gsi must correspond to a shared peripheral interrupt (SPI), ie the GIC interrupt ID is gsi + 32. It is still under discussion whether PPI injection support is needed. this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD. No IRQ routing table is used. 2 patch files are included: - the 1st one simply removes the inclusion of irq.h. After Paul Mackerras' eventfd rework, I think it is no more needed - the second patch brings the irqfd integration for ARM, without routing This patch serie deprecates integration with GSI routing (https://patches.linaro.org/32261/) can be found at git://git.linaro.org/people/eric.auger/linux.git on branch irqfd_integ_v5 This work was tested with Calxeda Midway xgmac main interrupt with qemu-system-arm and QEMU VFIO platform device. - rebase on 3.17rc1 - move of the dist unlock in process_maintenance - remove of dist lock in __kvm_vgic_sync_hwstate - remove irq.h Eric Auger (2): KVM: EVENTFD: remove inclusion of irq.h KVM: ARM: add irqfd support Documentation/virtual/kvm/api.txt | 5 +++- arch/arm/include/uapi/asm/kvm.h | 3 +++ arch/arm/kvm/Kconfig | 3 ++- arch/arm/kvm/Makefile | 2 +- virt/kvm/arm/vgic.c | 56 --- virt/kvm/eventfd.c| 1 - 6 files changed, 62 insertions(+), 8 deletions(-) -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 2/2] KVM: ARM: add irqfd support
This patch enables irqfd on ARM.
irqfd framework enables to inject a virtual IRQ into a guest upon an
eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
a kvm_irqfd struct that associates a VM, an eventfd, an IRQ number
(aka. the gsi). When an actor signals the eventfd (typically a VFIO
platform driver), the kvm irqfd subsystem injects the provided virtual
IRQ into the guest.
Resamplefd also is supported for level sensitive interrupts, ie. the
user can provide another eventfd that is triggered when the completion
of the virtual IRQ (gsi) is detected by the GIC.
The gsi must correspond to a shared peripheral interrupt (SPI), ie the
GIC interrupt ID is gsi+32.
this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
No IRQ routing table is used.
Signed-off-by: Eric Auger
---
This patch deprecates the previous patch featuring GSI routing
(https://patches.linaro.org/32261/)
irqchip.c and irq_comm.c are not used at all.
This RFC applies on top of Christoffer Dall's serie
arm/arm64: KVM: Various VGIC cleanups and improvements
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
branch irqfd_integ_v5
This work was tested with Calxeda Midway xgmac main interrupt with
qemu-system-arm and QEMU VFIO platform device.
v1 -> v2:
- rebase on 3.17rc1
- move of the dist unlock in process_maintenance
- remove of dist lock in __kvm_vgic_sync_hwstate
- rewording of the commit message (add resamplefd reference)
- remove irq.h
---
Documentation/virtual/kvm/api.txt | 5 +++-
arch/arm/include/uapi/asm/kvm.h | 3 +++
arch/arm/kvm/Kconfig | 3 ++-
arch/arm/kvm/Makefile | 2 +-
virt/kvm/arm/vgic.c | 56 ---
5 files changed, 62 insertions(+), 7 deletions(-)
diff --git a/Documentation/virtual/kvm/api.txt
b/Documentation/virtual/kvm/api.txt
index beae3fd..8118b12 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2204,7 +2204,7 @@ into the hash PTE second double word).
4.75 KVM_IRQFD
Capability: KVM_CAP_IRQFD
-Architectures: x86 s390
+Architectures: x86 s390 arm
Type: vm ioctl
Parameters: struct kvm_irqfd (in)
Returns: 0 on success, -1 on error
@@ -2230,6 +2230,9 @@ Note that closing the resamplefd is not sufficient to
disable the
irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
+On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
+This means the programmed GIC interrupt ID is gsi+32.
+
4.76 KVM_PPC_ALLOCATE_HTAB
Capability: KVM_CAP_PPC_ALLOC_HTAB
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index e6ebdd3..3034c66 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
/* Highest supported SPI, from VGIC_NR_IRQS */
#define KVM_ARM_IRQ_GIC_MAX127
+/* One single KVM irqchip, ie. the VGIC */
+#define KVM_NR_IRQCHIPS 1
+
/* PSCI interface */
#define KVM_PSCI_FN_BASE 0x95c1ba5e
#define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 466bd29..82ccd81 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -24,6 +24,7 @@ config KVM
select KVM_MMIO
select KVM_ARM_HOST
depends on ARM_VIRT_EXT && ARM_LPAE
+ select HAVE_KVM_EVENTFD
---help---
Support hosting virtualized guest machines. You will also
need to select one or more of the processor modules below.
@@ -55,7 +56,7 @@ config KVM_ARM_MAX_VCPUS
config KVM_ARM_VGIC
bool "KVM support for Virtual GIC"
depends on KVM_ARM_HOST && OF
- select HAVE_KVM_IRQCHIP
+ select HAVE_KVM_IRQFD
default y
---help---
Adds support for a hardware assisted, in-kernel GIC emulation.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index f7057ed..859db09 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
KVM := ../../../virt/kvm
-kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
+kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 7164d2e..586bd11 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1334,7 +1334,10 @@ epilog:
static bool vgic_process_maintenance(struct kvm_vcpu *vcpu)
{
u32 status = vgic_get_interrupt_status(vcpu);
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
bool level_pending
[PATCH v2 1/2] KVM: EVENTFD: remove inclusion of irq.h
No more needed. Also irq.h is not used on ARM. Signed-off-by: Eric Auger --- virt/kvm/eventfd.c | 1 - 1 file changed, 1 deletion(-) diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 3c5981c..0c712a7 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -36,7 +36,6 @@ #include #include -#include "irq.h" #include "iodev.h" #ifdef CONFIG_HAVE_KVM_IRQFD -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 9/9] KVM: KVM_VFIO: ARM: implement irq forwarding control
Implements ARM specific KVM-VFIO device group commands:
- KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ
- KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ
capability can be queried using KVM_HAS_DEVICE_ATTR.
The new commands enable to set IRQ forwarding on/off for a given
IRQ index of a VFIO platform device.
as soon as a forwarded irq is set, a reference to the VFIO device
is taken by the kvm-vfio device.
The kvm-vfio device stores in the kvm_vfio_arch_data the list
of "assigned" devices (kvm_vfio_device). Each kvm_vfio_device
stores the list of assigned IRQs (potentially allowed a subset of
IRQ to be forwarded)
The kvm-vfio device programs both the GIC and vGIC. Also it
clears the active bit on destruction, in case the guest did not
do it itself.
Changing the forwarded state is not allowed in the critical
section starting from VFIO IRQ handler to LR programming. It is
up to the client to take care of this.
Signed-off-by: Eric Auger
---
arch/arm/include/asm/kvm_host.h | 2 +
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/kvm_vfio_arm.c | 599
3 files changed, 602 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/kvm/kvm_vfio_arm.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 4f1edbf..5c300f6 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -25,6 +25,8 @@
#include
#include
+#define __KVM_HAVE_ARCH_KVM_VFIO
+
#if defined(CONFIG_KVM_ARM_MAX_VCPUS)
#define KVM_MAX_VCPUS CONFIG_KVM_ARM_MAX_VCPUS
#else
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index ea1fa76..26a5a42 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -19,7 +19,7 @@ kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
$(KVM)/eventfd.o $(KVM)/vf
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
-obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o
+obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o kvm_vfio_arm.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic-v2.o
obj-$(CONFIG_KVM_ARM_TIMER) += $(KVM)/arm/arch_timer.o
diff --git a/arch/arm/kvm/kvm_vfio_arm.c b/arch/arm/kvm/kvm_vfio_arm.c
new file mode 100644
index 000..6619e0b
--- /dev/null
+++ b/arch/arm/kvm/kvm_vfio_arm.c
@@ -0,0 +1,599 @@
+/*
+ * Copyright (C) 2014 Linaro Ltd.
+ * Authors: Eric Auger
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+struct vfio_device;
+
+enum kvm_irq_fwd_action {
+ KVM_VFIO_IRQ_SET_FORWARD,
+ KVM_VFIO_IRQ_SET_NORMAL,
+ KVM_VFIO_IRQ_CLEANUP,
+};
+
+/* internal structure describing a forwarded IRQ */
+struct __kvm_arch_fwd_irq {
+ struct list_head link;
+ __u32 irq_index; /* platform device irq index */
+ __u32 hwirq; /*physical IRQ */
+ __u32 guest_irq; /* virtual IRQ */
+ struct kvm_vcpu *vcpu; /* vcpu to inject into*/
+};
+
+struct kvm_vfio_device {
+ struct list_head node;
+ struct vfio_device *vfio_device;
+ /* list of forwarded IRQs for that VFIO device */
+ struct list_head fwd_irq_list;
+ int fd;
+};
+
+struct kvm_vfio_arch_data {
+ /* list of kvm_vfio_devices for which some IRQs are forwarded*/
+ struct list_head assigned_device_list;
+};
+
+/**
+ * set_fwd_state - change the forwarded state of an IRQ
+ * @pfwd: the forwarded irq struct
+ * @action: action to perform (set forward, set back normal, cleanup)
+ *
+ * programs the GIC and VGIC
+ * returns the VGIC map/unmap return status
+ * It is the responsability of the caller to make sure the physical IRQ
+ * is not active. there is a critical section between the start of the
+ * VFIO IRQ handler and LR programming.
+ */
+int set_fwd_state(struct __kvm_arch_fwd_irq *pfwd,
+ enum kvm_irq_fwd_action action)
+{
+ int ret;
+ struct irq_desc *desc = irq_to_desc(pfwd->hwirq);
+ struct irq_data *d = &desc->irq_data;
+ struct irq_chip *chip = desc->irq_data.chip;
+
+ disable_irq(pfwd->hwirq);
+ /* no fwd state change can happen if the IRQ is in progress */
+ if (irqd_irq_inprogress(d)) {
+ kvm_err("%s cannot change fwd state (IRQ %d in progress\n",
+ __func__, pfwd->hwirq);
+ enable_irq(pfwd->hwirq);
+ return -1;
+ }
+
+
[RFC 3/9] VFIO: platform: handler tests whether the IRQ is forwarded
In case the IRQ is forwarded, the VFIO platform IRQ handler does not
need to disable the IRQ anymore. In that mode, when the handler completes
the IRQ is not deactivated but only its priority is lowered.
Some other actor (typically a guest) is supposed to deactivate the IRQ,
allowing at that time a new physical IRQ to hit.
In virtualization use case, the physical IRQ is automatically completed
by the interrupt controller when the guest completes the corresponding
virtual IRQ.
Signed-off-by: Eric Auger
---
drivers/vfio/platform/vfio_platform_irq.c | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vfio/platform/vfio_platform_irq.c
b/drivers/vfio/platform/vfio_platform_irq.c
index 6768508..1f851b2 100644
--- a/drivers/vfio/platform/vfio_platform_irq.c
+++ b/drivers/vfio/platform/vfio_platform_irq.c
@@ -88,13 +88,18 @@ static irqreturn_t vfio_irq_handler(int irq, void *dev_id)
struct vfio_platform_irq *irq_ctx = dev_id;
unsigned long flags;
int ret = IRQ_NONE;
+ struct irq_data *d;
+ bool is_forwarded;
spin_lock_irqsave(&irq_ctx->lock, flags);
if (!irq_ctx->masked) {
ret = IRQ_HANDLED;
+ d = irq_get_irq_data(irq_ctx->hwirq);
+ is_forwarded = irqd_irq_forwarded(d);
- if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED) {
+ if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED &&
+ !is_forwarded) {
disable_irq_nosync(irq_ctx->hwirq);
irq_ctx->masked = true;
}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 2/9] KVM: ARM: VGIC: add forwarded irq rbtree lock
add a lock related to the rb tree manipulation. The rb tree can be
searched in one thread (irqfd handler for instance) and map/unmap
happen in another.
Signed-off-by: Eric Auger
---
virt/kvm/arm/vgic.c | 46 +-
1 file changed, 37 insertions(+), 9 deletions(-)
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 195c10c..3311e0a 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1628,9 +1628,15 @@ static struct rb_root *vgic_get_irq_phys_map(struct
kvm_vcpu *vcpu,
int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct rb_root *root = vgic_get_irq_phys_map(vcpu, virt_irq);
- struct rb_node **new = &root->rb_node, *parent = NULL;
+ struct rb_root *root;
+ struct rb_node **new, *parent = NULL;
struct irq_phys_map *new_map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ spin_lock(&dist->rb_tree_lock);
+
+ root = vgic_get_irq_phys_map(vcpu, virt_irq);
+ new = &root->rb_node;
/* Boilerplate rb_tree code */
while (*new) {
@@ -1642,13 +1648,17 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
virt_irq, int phys_irq)
new = &(*new)->rb_left;
else if (this->virt_irq > virt_irq)
new = &(*new)->rb_right;
- else
+ else {
+ spin_unlock(&dist->rb_tree_lock);
return -EEXIST;
+ }
}
new_map = kzalloc(sizeof(*new_map), GFP_KERNEL);
- if (!new_map)
+ if (!new_map) {
+ spin_unlock(&dist->rb_tree_lock);
return -ENOMEM;
+ }
new_map->virt_irq = virt_irq;
new_map->phys_irq = phys_irq;
@@ -1656,6 +1666,8 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
virt_irq, int phys_irq)
rb_link_node(&new_map->node, parent, new);
rb_insert_color(&new_map->node, root);
+ spin_unlock(&dist->rb_tree_lock);
+
return 0;
}
@@ -1683,24 +1695,39 @@ static struct irq_phys_map *vgic_irq_map_search(struct
kvm_vcpu *vcpu,
int vgic_get_phys_irq(struct kvm_vcpu *vcpu, int virt_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+ int ret;
+
+ spin_lock(&dist->rb_tree_lock);
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map)
- return map->phys_irq;
+ ret = map->phys_irq;
+ else
+ ret = -ENOENT;
+
+ spin_unlock(&dist->rb_tree_lock);
+ return ret;
- return -ENOENT;
}
int vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ spin_lock(&dist->rb_tree_lock);
+
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map && map->phys_irq == phys_irq) {
rb_erase(&map->node, vgic_get_irq_phys_map(vcpu, virt_irq));
kfree(map);
+ spin_unlock(&dist->rb_tree_lock);
return 0;
}
-
+ spin_unlock(&dist->rb_tree_lock);
return -ENOENT;
}
@@ -1896,6 +1923,7 @@ int kvm_vgic_create(struct kvm *kvm)
}
spin_lock_init(&kvm->arch.vgic.lock);
+ spin_lock_init(&kvm->arch.vgic.rb_tree_lock);
kvm->arch.vgic.in_kernel = true;
kvm->arch.vgic.vctrl_base = vgic->vctrl_base;
kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 0/9] KVM-VFIO IRQ forward control
This RFC proposes an integration of "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) in KVM. It enables to transform a VFIO platform driver IRQ into a forwarded IRQ. The direct benefit is that, for a level sensitive IRQ, a VM switch can be avoided on guest virtual IRQ completion. Before this patch, a maintenance IRQ was triggered on the virtual IRQ completion. When the IRQ is forwarded, the VFIO platform driver does not need to disable the IRQ anymore. Indeed when returning from the IRQ handler the IRQ is not deactivated. Only its priority is lowered. This means the same IRQ cannot hit before the guest completes the virtual IRQ and the GIC automatically deactivates the corresponding physical IRQ. Besides, the injection still is based on irqfd triggering. The only impact on irqfd process is resamplefd is not called anymore on virtual IRQ completion since this latter becomes "transparent". The current integration is based on an extension of the KVM-VFIO device, previously used by KVM to interact with VFIO groups. The patch serie now enables KVM to directly interact with a VFIO platform device. The VFIO external API was extended for that purpose. Th KVM-VFIO device can get/put the vfio platform device, check its integrity and type, get the IRQ number associated to an IRQ index. The KVM-VFIO is extended with an architecture specific implementation. IRQ forward control is implemented in the ARM specific part. from a user point of view, the functionality is provided through new KVM-VFIO device commands, KVM_DEV_VFIO_DEVICE_(DE)ASSIGN_IRQ and the capability can be checked with KVM_HAS_DEVICE_ATTR. Assignment can only be changed when the physical IRQ is not active. It is the responsability of the user to do this check. This patch serie has the following dependencies: - "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) in - [PATCH v2] irqfd for ARM which itself depends on - arm/arm64: KVM: Various VGIC cleanups and improvements http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/263685.html - and obviously the VFIO platform driver serie: [RFC PATCH v6 00/20] VFIO support for platform devices on ARM https://www.mail-archive.com/kvm@vger.kernel.org/msg103247.html Integrated pieces can be found at git://git.linaro.org/people/eric.auger/linux.git on branch 3.17rc1_forward_integ_v0 This was was tested on Calxeda Miday, assigning the xgmac main IRQ. Eric Auger (9): KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded IRQ KVM: ARM: VGIC: add forwarded irq rbtree lock VFIO: platform: handler tests whether the IRQ is forwarded KVM: KVM-VFIO: update user API to program forwarded IRQ VFIO: Extend external user API KVM: KVM-VFIO: allow arch specific implementation KVM: KVM-VFIO: add new VFIO external API hooks KVM: KVM-VFIO: add kvm_vfio_arch_data and accessors KVM: KVM_VFIO: ARM: implement irq forwarding control Documentation/virtual/kvm/devices/vfio.txt | 25 ++ arch/arm/include/asm/kvm_host.h| 16 + arch/arm/include/uapi/asm/kvm.h| 6 + arch/arm/kvm/Makefile | 2 +- arch/arm/kvm/kvm_vfio_arm.c| 599 + drivers/vfio/platform/vfio_platform_irq.c | 7 +- drivers/vfio/vfio.c| 35 ++ include/kvm/arm_vgic.h | 1 + include/linux/kvm_host.h | 30 ++ include/linux/vfio.h | 4 + include/uapi/linux/kvm.h | 3 + virt/kvm/arm/vgic.c| 55 ++- virt/kvm/vfio.c| 92 + 13 files changed, 862 insertions(+), 13 deletions(-) create mode 100644 arch/arm/kvm/kvm_vfio_arm.c -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 8/9] KVM: KVM-VFIO: add kvm_vfio_arch_data and accessors
add a pointer to architecture specific data in kvm_vfio struct
add accessors to keep kvm_vfio private
Signed-off-by: Eric Auger
---
arch/arm/include/asm/kvm_host.h | 8
virt/kvm/vfio.c | 21 +
2 files changed, 29 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 62cbf5b..4f1edbf 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -177,6 +177,14 @@ void kvm_vfio_device_put_external_user(struct vfio_device
*vdev);
int kvm_vfio_external_get_type(struct vfio_device *vdev);
struct device *kvm_vfio_external_get_base_device(struct vfio_device *vdev);
+struct kvm_vfio;
+struct kvm_vfio_arch_data;
+void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
+ struct kvm_vfio_arch_data *ptr);
+struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio *kv);
+void kvm_vfio_lock(struct kvm_vfio *kv);
+void kvm_vfio_unlock(struct kvm_vfio *kv);
+
/* We do not have shadow page tables, hence the empty hooks */
static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
{
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index f1c4e35..177b71e 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -28,6 +28,7 @@ struct kvm_vfio {
struct list_head group_list;
struct mutex lock;
bool noncoherent;
+ struct kvm_vfio_arch_data *arch_data;
};
static struct vfio_group *kvm_vfio_group_get_external_user(struct file *filep)
@@ -338,6 +339,26 @@ static int kvm_vfio_create(struct kvm_device *dev, u32
type)
return 0;
}
+void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
+ struct kvm_vfio_arch_data *ptr)
+{
+ kv->arch_data = ptr;
+}
+
+struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio *kv)
+{
+ return kv->arch_data;
+}
+
+void kvm_vfio_lock(struct kvm_vfio *kv)
+{
+ mutex_lock(&kv->lock);
+}
+
+void kvm_vfio_unlock(struct kvm_vfio *kv)
+{
+ mutex_unlock(&kv->lock);
+}
struct kvm_device_ops kvm_vfio_ops = {
.name = "kvm-vfio",
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 1/9] KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded IRQ
Fix multiple injection of level sensitive forwarded IRQs.
With current code, the second injection fails since the state bitmaps
are not reset (process_maintenance is not called anymore).
New implementation consists in fully bypassing the vgic state
management for forwarded IRQ (checks are ignored in
vgic_update_irq_pending). This obviously assumes the forwarded IRQ is
injected from kernel side.
---
It was attempted to reset the states in __kvm_vgic_sync_hwstate, checking
the emptied LR of forwarded IRQ. However surprisingly this solution does
not seem to work. Some times, a new forwarded IRQ injection is observed
while the LR of the previous instance was not observed as empty.
Signed-off-by: Eric Auger
---
include/kvm/arm_vgic.h | 1 +
virt/kvm/arm/vgic.c| 9 +++--
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 743020f..3da244f 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -177,6 +177,7 @@ struct vgic_dist {
unsigned long irq_pending_on_cpu;
struct rb_root irq_phys_map;
+ spinlock_t rb_tree_lock;
#endif
};
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 0007300..195c10c 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1517,14 +1517,18 @@ static bool vgic_update_irq_pending(struct kvm *kvm,
int cpuid,
int edge_triggered, level_triggered;
int enabled;
bool ret = true;
+ bool is_forwarded;
spin_lock(&dist->lock);
vcpu = kvm_get_vcpu(kvm, cpuid);
+ is_forwarded = (vgic_get_phys_irq(vcpu, irq_num) >0);
+
edge_triggered = vgic_irq_is_edge(vcpu, irq_num);
level_triggered = !edge_triggered;
- if (!vgic_validate_injection(vcpu, irq_num, level)) {
+ if (!is_forwarded &&
+ !vgic_validate_injection(vcpu, irq_num, level)) {
ret = false;
goto out;
}
@@ -1557,7 +1561,8 @@ static bool vgic_update_irq_pending(struct kvm *kvm, int
cpuid,
goto out;
}
- if (level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
+ if (!is_forwarded &&
+ level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
/*
* Level interrupt in progress, will be picked up
* when EOId.
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 7/9] KVM: KVM-VFIO: add new VFIO external API hooks
add functions that implement the gateway to the extended
external VFIO API:
- kvm_vfio_device_get_external_user
- kvm_vfio_device_put_external_user
- kvm_vfio_external_get_type
- kvm_vfio_external_get_base_device
Signed-off-by: Eric Auger
---
arch/arm/include/asm/kvm_host.h | 6
virt/kvm/vfio.c | 62 +
2 files changed, 68 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 6dfb404..62cbf5b 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -171,6 +171,12 @@ void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva,
pte_t pte);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
+struct vfio_device;
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep);
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev);
+int kvm_vfio_external_get_type(struct vfio_device *vdev);
+struct device *kvm_vfio_external_get_base_device(struct vfio_device *vdev);
+
/* We do not have shadow page tables, hence the empty hooks */
static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
{
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index 89d3b75..f1c4e35 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -59,6 +59,67 @@ static void kvm_vfio_group_put_external_user(struct
vfio_group *vfio_group)
symbol_put(vfio_group_put_external_user);
}
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev;
+ struct vfio_device *(*fn)(struct file *);
+
+ fn = symbol_get(vfio_device_get_external_user);
+ if (!fn)
+ return ERR_PTR(-EINVAL);
+
+ vdev = fn(filep);
+
+ symbol_put(vfio_device_get_external_user);
+
+ return vdev;
+}
+
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ void (*fn)(struct vfio_device *);
+
+ fn = symbol_get(vfio_device_put_external_user);
+ if (!fn)
+ return;
+
+ fn(vdev);
+
+ symbol_put(vfio_device_put_external_user);
+}
+
+int kvm_vfio_external_get_type(struct vfio_device *vdev)
+{
+ int (*fn)(struct vfio_device *);
+ int ret;
+
+ fn = symbol_get(vfio_external_get_type);
+ if (!fn)
+ return -EINVAL;
+
+ ret = fn(vdev);
+
+ symbol_put(vfio_external_get_type);
+
+ return ret;
+}
+
+struct device *kvm_vfio_external_get_base_device(struct vfio_device *vdev)
+{
+ struct device *(*fn)(struct vfio_device *);
+ struct device *dev;
+
+ fn = symbol_get(vfio_external_get_base_device);
+ if (!fn)
+ return NULL;
+
+ dev = fn(vdev);
+
+ symbol_put(vfio_external_get_base_device);
+
+ return dev;
+}
+
static bool kvm_vfio_group_is_coherent(struct vfio_group *vfio_group)
{
long (*fn)(struct vfio_group *, unsigned long);
@@ -277,6 +338,7 @@ static int kvm_vfio_create(struct kvm_device *dev, u32 type)
return 0;
}
+
struct kvm_device_ops kvm_vfio_ops = {
.name = "kvm-vfio",
.create = kvm_vfio_create,
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 5/9] VFIO: Extend external user API
New functions are added to be called from ARM KVM-VFIO device.
- vfio_device_get_external_user enables to get a vfio device from
its fd
- vfio_device_put_external_user puts the vfio device
- vfio_external_get_type enables to retrieve the type of the device
(PCI or platform)
- vfio_external_get_base_device enables to get the
struct device*, useful to access the platform_device
Signed-off-by: Eric Auger
---
drivers/vfio/vfio.c | 35 +++
include/linux/vfio.h | 4
2 files changed, 39 insertions(+)
diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index 8e84471..c93b9e4 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1401,6 +1401,41 @@ void vfio_group_put_external_user(struct vfio_group
*group)
}
EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
+struct vfio_device *vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev = filep->private_data;
+
+ if (filep->f_op != &vfio_device_fops)
+ return ERR_PTR(-EINVAL);
+
+ vfio_device_get(vdev);
+ return vdev;
+}
+EXPORT_SYMBOL_GPL(vfio_device_get_external_user);
+
+void vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ vfio_device_put(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_device_put_external_user);
+
+int vfio_external_get_type(struct vfio_device *vdev)
+{
+ if (!strcmp(vdev->ops->name, "vfio-platform"))
+ return VFIO_DEVICE_FLAGS_PLATFORM;
+ else if (!strcmp(vdev->ops->name, "vfio-pci"))
+ return VFIO_DEVICE_FLAGS_PCI;
+ else
+ return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(vfio_external_get_type);
+
+struct device *vfio_external_get_base_device(struct vfio_device *vdev)
+{
+ return vdev->dev;
+}
+EXPORT_SYMBOL_GPL(vfio_external_get_base_device);
+
int vfio_external_user_iommu_id(struct vfio_group *group)
{
return iommu_group_id(group->iommu_group);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index ffe04ed..19e98eb 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -99,6 +99,10 @@ extern void vfio_group_put_external_user(struct vfio_group
*group);
extern int vfio_external_user_iommu_id(struct vfio_group *group);
extern long vfio_external_check_extension(struct vfio_group *group,
unsigned long arg);
+extern struct vfio_device *vfio_device_get_external_user(struct file *filep);
+extern void vfio_device_put_external_user(struct vfio_device *vdev);
+extern int vfio_external_get_type(struct vfio_device *vdev);
+extern struct device *vfio_external_get_base_device(struct vfio_device *vdev);
struct pci_dev;
#ifdef CONFIG_EEH
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 6/9] KVM: KVM-VFIO: allow arch specific implementation
introduce a new option __KVM_HAVE_ARCH_KVM_VFIO option.
When set the generic KVM-VFIO code calls architecture dependent
code.
the architecture dependent hooks are
- kvm_arch_vfio_has_attr
- kvm_arch_vfio_set_attr
- kvm_arch_vfio_init
- kvm_arch_vfio_destroy
Signed-off-by: Eric Auger
---
include/linux/kvm_host.h | 30 ++
virt/kvm/vfio.c | 9 +
2 files changed, 39 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index a4c33b3..c4ce4af 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1075,6 +1075,36 @@ extern struct kvm_device_ops kvm_vfio_ops;
extern struct kvm_device_ops kvm_arm_vgic_v2_ops;
extern struct kvm_device_ops kvm_flic_ops;
+#ifdef __KVM_HAVE_ARCH_KVM_VFIO
+
+int kvm_arch_vfio_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr);
+int kvm_arch_vfio_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr);
+int kvm_arch_vfio_init(struct kvm_device *dev);
+
+void kvm_arch_vfio_destroy(struct kvm_device *dev);
+
+#else
+static inline int kvm_arch_vfio_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+static inline int kvm_arch_vfio_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+static inline int kvm_arch_vfio_init(struct kvm_device *dev)
+{
+ return 0;
+}
+static inline void kvm_arch_vfio_destroy(struct kvm_device *dev)
+{
+}
+#endif
+
#ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
static inline void kvm_vcpu_set_in_spin_loop(struct kvm_vcpu *vcpu, bool val)
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index ba1a93f..89d3b75 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -207,6 +207,8 @@ static int kvm_vfio_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_VFIO_GROUP:
return kvm_vfio_set_group(dev, attr->attr, attr->addr);
+ default:
+ return kvm_arch_vfio_set_attr(dev, attr);
}
return -ENXIO;
@@ -224,6 +226,9 @@ static int kvm_vfio_has_attr(struct kvm_device *dev,
}
break;
+
+ default:
+ kvm_arch_vfio_has_attr(dev, attr);
}
return -ENXIO;
@@ -234,6 +239,8 @@ static void kvm_vfio_destroy(struct kvm_device *dev)
struct kvm_vfio *kv = dev->private;
struct kvm_vfio_group *kvg, *tmp;
+ kvm_arch_vfio_destroy(dev);
+
list_for_each_entry_safe(kvg, tmp, &kv->group_list, node) {
kvm_vfio_group_put_external_user(kvg->vfio_group);
list_del(&kvg->node);
@@ -265,6 +272,8 @@ static int kvm_vfio_create(struct kvm_device *dev, u32 type)
dev->private = kv;
+ kvm_arch_vfio_init(dev);
+
return 0;
}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 4/9] KVM: KVM-VFIO: update user API to program forwarded IRQ
add new device group commands:
- KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ and
KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ
which enable to turn forwarded IRQ mode on/off.
Signed-off-by: Eric Auger
---
Documentation/virtual/kvm/devices/vfio.txt | 25 +
arch/arm/include/uapi/asm/kvm.h| 6 ++
include/uapi/linux/kvm.h | 3 +++
3 files changed, 34 insertions(+)
diff --git a/Documentation/virtual/kvm/devices/vfio.txt
b/Documentation/virtual/kvm/devices/vfio.txt
index ef51740..c8b3fa1 100644
--- a/Documentation/virtual/kvm/devices/vfio.txt
+++ b/Documentation/virtual/kvm/devices/vfio.txt
@@ -13,6 +13,7 @@ VFIO-group is held by KVM.
Groups:
KVM_DEV_VFIO_GROUP
+ KVM_DEV_VFIO_DEVICE
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
@@ -20,3 +21,27 @@ KVM_DEV_VFIO_GROUP attributes:
For each, kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
+
+KVM_DEV_VFIO_DEVICE attributes:
+ KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ
+ KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ
+
+For each, kvm_device_attr.addr points to an kvm_arch_forwarded_irq.
+This user API makes possible to create a special IRQ handling mode,
+currently supported only on ARM, where KVM and a VFIO platform driver
+collaborate to improve IRQ handling performance.
+fd represents the file descriptor of a valid VFIO device whose physical
+IRQ, referenced by its irq_index is injected to the VM guest_irq.
+
+On ASSIGN_IRQ, KVM-VFIO device programs:
+- the host, to not complete the physical IRQ itself.
+- the GIC, to automatically complete the physical IRQ when the guest
+ completes the virtual IRQ
+This avoid trapping the end-of-interrupt for level sensitive IRQ.
+
+On DEASSIGN_IRQ, one come back to the mode where the host completes the
+physical IRQ and the guest only completes the virtual IRQ.
+
+It is up to the caller of this API to get the assurance the IRQ is not
+outstanding when the ASSIGN/DEASSIGN is called. This could lead to some
+inconsistency on who is going to complete the IRQ.
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 3034c66..1920b33 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -109,6 +109,12 @@ struct kvm_sync_regs {
struct kvm_arch_memory_slot {
};
+struct kvm_arch_forwarded_irq {
+ __u32 fd; /* file desciptor of the VFIO device */
+ __u32 irq_index; /* platform device index of the IRQ */
+ __u32 guest_irq; /* virtual IRQ number */
+};
+
/* If you need to interpret the index values, here is the key: */
#define KVM_REG_ARM_COPROC_MASK0x0FFF
#define KVM_REG_ARM_COPROC_SHIFT 16
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index cf3a2ff..b149ba8 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -954,6 +954,9 @@ struct kvm_device_attr {
#define KVM_DEV_VFIO_GROUP1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_DEVICE 2
+#define KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ 1
+#define KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ 2
#define KVM_DEV_TYPE_ARM_VGIC_V2 5
#define KVM_DEV_TYPE_FLIC 6
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 0/9] KVM-VFIO IRQ forward control
On 08/26/2014 07:49 PM, Alex Williamson wrote: > On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote: >> This RFC proposes an integration of "ARM: Forwarding physical >> interrupts to a guest VM" (http://lwn.net/Articles/603514/) in >> KVM. >> >> It enables to transform a VFIO platform driver IRQ into a forwarded >> IRQ. The direct benefit is that, for a level sensitive IRQ, a VM >> switch can be avoided on guest virtual IRQ completion. Before this >> patch, a maintenance IRQ was triggered on the virtual IRQ completion. >> >> When the IRQ is forwarded, the VFIO platform driver does not need to >> disable the IRQ anymore. Indeed when returning from the IRQ handler >> the IRQ is not deactivated. Only its priority is lowered. This means >> the same IRQ cannot hit before the guest completes the virtual IRQ >> and the GIC automatically deactivates the corresponding physical IRQ. >> >> Besides, the injection still is based on irqfd triggering. The only >> impact on irqfd process is resamplefd is not called anymore on >> virtual IRQ completion since this latter becomes "transparent". >> >> The current integration is based on an extension of the KVM-VFIO >> device, previously used by KVM to interact with VFIO groups. The >> patch serie now enables KVM to directly interact with a VFIO >> platform device. The VFIO external API was extended for that purpose. >> >> Th KVM-VFIO device can get/put the vfio platform device, check its >> integrity and type, get the IRQ number associated to an IRQ index. >> >> The KVM-VFIO is extended with an architecture specific implementation. >> IRQ forward control is implemented in the ARM specific part. >> >> from a user point of view, the functionality is provided through new >> KVM-VFIO device commands, KVM_DEV_VFIO_DEVICE_(DE)ASSIGN_IRQ >> and the capability can be checked with KVM_HAS_DEVICE_ATTR. >> Assignment can only be changed when the physical IRQ is not active. >> It is the responsability of the user to do this check. >> >> This patch serie has the following dependencies: >> - "ARM: Forwarding physical interrupts to a guest VM" >> (http://lwn.net/Articles/603514/) in >> - [PATCH v2] irqfd for ARM >> which itself depends on >> - arm/arm64: KVM: Various VGIC cleanups and improvements >> >> http://lists.infradead.org/pipermail/linux-arm-kernel/2014-June/263685.html >> - and obviously the VFIO platform driver serie: >> [RFC PATCH v6 00/20] VFIO support for platform devices on ARM >> https://www.mail-archive.com/kvm@vger.kernel.org/msg103247.html >> >> Integrated pieces can be found at >> git://git.linaro.org/people/eric.auger/linux.git >> on branch 3.17rc1_forward_integ_v0 >> >> This was was tested on Calxeda Miday, assigning the xgmac main IRQ. > > Presumably this optimization should provide lower interrupt exit latency > and lower CPU overhead since we avoid the entire EOI path of the > resampler. Does it? It seems like there should be a measurable > improvement with something like netperf TCP_RR with this series. > Thanks, Hi Alex, I will publish some performance figures soon. I am currently missing a second node to run netserver. My preliminary understanding is perf improvement will come from 1) reduction of EOI latency 2) potential saving of VM switches, with 2) depending on thephysical IRQ rate. VERY HIGH RATE: Without the patch (traditional irqfd on ARM with maintenance IRQ): guest completes the vIRQ -> maintenance IRQ handler -> guest->host VM switch -> resamplefd trigger (virqfd) -> enable physical IRQ -> new physical IRQ hits -> VFIO handler -> fd trigger -> injection in guest -> host->guest VM switch with the patch guest completes the vIRQ -> GIC completes the physical IRQ -> new physical IRQ hits -> guest->host VM switch -> VFIO handler -> fd trigger -> injection in guest -> host->guest VM switch => Same number of VM switches SLOWER RATE: Without the patch: guest completes the vIRQ -> maintenance IRQ handler -> guest->host VM switch -> resamplefd trigger (virqfd) -> enable physical IRQ [host ..] host->guest VM switch [guest ..] physical IRQ hits -> guest->host VM switch -> VFIO handler -> fd trigger -> injection in guest -> host->guest VM switch With that patch: guest completes the vIRQ -> GIC completes the physical IRQ [guest ..] physical IRQ hits -> guest->host VM switch -> VFIO handler -> fd trigger -> injection in guest -> host->guest VM switch Hence less VM switches with that patch. But it is also related to scheduling, relative load of host/gues
Re: [RFC 9/9] KVM: KVM_VFIO: ARM: implement irq forwarding control
On 08/26/2014 09:02 PM, Alex Williamson wrote: > On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote: >> Implements ARM specific KVM-VFIO device group commands: >> - KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ >> - KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ >> capability can be queried using KVM_HAS_DEVICE_ATTR. >> >> The new commands enable to set IRQ forwarding on/off for a given >> IRQ index of a VFIO platform device. >> >> as soon as a forwarded irq is set, a reference to the VFIO device >> is taken by the kvm-vfio device. >> >> The kvm-vfio device stores in the kvm_vfio_arch_data the list >> of "assigned" devices (kvm_vfio_device). Each kvm_vfio_device >> stores the list of assigned IRQs (potentially allowed a subset of >> IRQ to be forwarded) >> >> The kvm-vfio device programs both the GIC and vGIC. Also it >> clears the active bit on destruction, in case the guest did not >> do it itself. >> >> Changing the forwarded state is not allowed in the critical >> section starting from VFIO IRQ handler to LR programming. It is >> up to the client to take care of this. >> >> Signed-off-by: Eric Auger >> --- >> arch/arm/include/asm/kvm_host.h | 2 + >> arch/arm/kvm/Makefile | 2 +- >> arch/arm/kvm/kvm_vfio_arm.c | 599 >> >> 3 files changed, 602 insertions(+), 1 deletion(-) >> create mode 100644 arch/arm/kvm/kvm_vfio_arm.c > > I'm really happy that it seems like the kvm-vfio device is going to work > for you, but I think too much stuff is being pushed out to arch code > here. Exporting the interfaces in patches 7 & 8 are setting the stage > for duplicate code for anyone wanting to implement device attributes. > Instead, I think the core code should support the list of > kvm_vfio_devices with proper cleanup, and we should attempt to access > the kvm_vfio_ callbacks as little as possible from arch code. Thanks, OK. my next iteration will feature much more generic code. Thanks for the review Best Regards Eric > > Alex > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 4/9] KVM: KVM-VFIO: update user API to program forwarded IRQ
On 08/26/2014 09:01 PM, Alex Williamson wrote:
> On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote:
>> add new device group commands:
>> - KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ and
>> KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ
>>
>> which enable to turn forwarded IRQ mode on/off.
>>
>> Signed-off-by: Eric Auger
>> ---
>> Documentation/virtual/kvm/devices/vfio.txt | 25 +
>> arch/arm/include/uapi/asm/kvm.h| 6 ++
>> include/uapi/linux/kvm.h | 3 +++
>> 3 files changed, 34 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
>> b/Documentation/virtual/kvm/devices/vfio.txt
>> index ef51740..c8b3fa1 100644
>> --- a/Documentation/virtual/kvm/devices/vfio.txt
>> +++ b/Documentation/virtual/kvm/devices/vfio.txt
>> @@ -13,6 +13,7 @@ VFIO-group is held by KVM.
>>
>> Groups:
>>KVM_DEV_VFIO_GROUP
>> + KVM_DEV_VFIO_DEVICE
>>
>> KVM_DEV_VFIO_GROUP attributes:
>>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
>> @@ -20,3 +21,27 @@ KVM_DEV_VFIO_GROUP attributes:
>>
>> For each, kvm_device_attr.addr points to an int32_t file descriptor
>> for the VFIO group.
>> +
>> +KVM_DEV_VFIO_DEVICE attributes:
>> + KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ
>> + KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ
>> +
>> +For each, kvm_device_attr.addr points to an kvm_arch_forwarded_irq.
>> +This user API makes possible to create a special IRQ handling mode,
>> +currently supported only on ARM, where KVM and a VFIO platform driver
>> +collaborate to improve IRQ handling performance.
>> +fd represents the file descriptor of a valid VFIO device whose physical
>> +IRQ, referenced by its irq_index is injected to the VM guest_irq.
>> +
>> +On ASSIGN_IRQ, KVM-VFIO device programs:
>> +- the host, to not complete the physical IRQ itself.
>> +- the GIC, to automatically complete the physical IRQ when the guest
>> + completes the virtual IRQ
>> +This avoid trapping the end-of-interrupt for level sensitive IRQ.
>> +
>> +On DEASSIGN_IRQ, one come back to the mode where the host completes the
>> +physical IRQ and the guest only completes the virtual IRQ.
>> +
>> +It is up to the caller of this API to get the assurance the IRQ is not
>> +outstanding when the ASSIGN/DEASSIGN is called. This could lead to some
>> +inconsistency on who is going to complete the IRQ.
>
> Why not call these FORWARD/UNFORWARD or something since the operation
> isn't really doing anything with assignment of the IRQ. The IRQ is
> already "assigned", we're modifying the behavior.
Sure I will change the name.
>
>> diff --git a/arch/arm/include/uapi/asm/kvm.h
>> b/arch/arm/include/uapi/asm/kvm.h
>> index 3034c66..1920b33 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -109,6 +109,12 @@ struct kvm_sync_regs {
>> struct kvm_arch_memory_slot {
>> };
>>
>> +struct kvm_arch_forwarded_irq {
>> +__u32 fd; /* file desciptor of the VFIO device */
>> +__u32 irq_index; /* platform device index of the IRQ */
>
> The vfio-platform device IRQ index? ARM is the only implementation we
> have of this, but to keep it generic maybe the comment should read "vfio
> device IRQ index".
I will replace by vfio device IRQ index.
>
>> +__u32 guest_irq; /* virtual IRQ number */
>
> This would be a GSI or similar concept if we were on x86.
ok for GSI now this naming seems better understood by the ARM community
too;-)
I'm a little
> confused that were using an arch structure here rather than trying to
> keep the kvm-vfio device interface neutral.
I will move much more things on the generic side then then assuming that
someone in the future may use such functionality. Actually the only ARM
specific implementation is the GIC interrupt controller programming. As
far as I see the rest is generic, in terms of API.
Thanks
Eric
Maybe it makes sense, I'm
> not sure. Thanks,
>
> Alex
>
>> +};
>> +
>> /* If you need to interpret the index values, here is the key: */
>> #define KVM_REG_ARM_COPROC_MASK 0x0FFF
>> #define KVM_REG_ARM_COPROC_SHIFT16
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index cf3a2ff..b149ba8 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -954,6 +954,9 @@ struct kvm_device_attr {
>> #define KVM_DEV_VFIO_GROUP 1
>> #define KVM_DEV_VFIO_GROUP_ADD1
>> #define KVM_DEV_VFIO_GROUP_DEL2
>> +#define KVM_DEV_VFIO_DEVICE2
>> +#define KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ1
>> +#define KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ 2
>> #define KVM_DEV_TYPE_ARM_VGIC_V25
>> #define KVM_DEV_TYPE_FLIC 6
>>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 8/9] KVM: KVM-VFIO: add kvm_vfio_arch_data and accessors
On 08/26/2014 09:02 PM, Alex Williamson wrote:
> On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote:
>> add a pointer to architecture specific data in kvm_vfio struct
>> add accessors to keep kvm_vfio private
>>
>> Signed-off-by: Eric Auger
>> ---
>> arch/arm/include/asm/kvm_host.h | 8
>> virt/kvm/vfio.c | 21 +
>> 2 files changed, 29 insertions(+)
>>
>> diff --git a/arch/arm/include/asm/kvm_host.h
>> b/arch/arm/include/asm/kvm_host.h
>> index 62cbf5b..4f1edbf 100644
>> --- a/arch/arm/include/asm/kvm_host.h
>> +++ b/arch/arm/include/asm/kvm_host.h
>> @@ -177,6 +177,14 @@ void kvm_vfio_device_put_external_user(struct
>> vfio_device *vdev);
>> int kvm_vfio_external_get_type(struct vfio_device *vdev);
>> struct device *kvm_vfio_external_get_base_device(struct vfio_device *vdev);
>>
>> +struct kvm_vfio;
>> +struct kvm_vfio_arch_data;
>> +void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
>> + struct kvm_vfio_arch_data *ptr);
>> +struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio
>> *kv);
>> +void kvm_vfio_lock(struct kvm_vfio *kv);
>> +void kvm_vfio_unlock(struct kvm_vfio *kv);
>> +
>> /* We do not have shadow page tables, hence the empty hooks */
>> static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
>> {
>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>> index f1c4e35..177b71e 100644
>> --- a/virt/kvm/vfio.c
>> +++ b/virt/kvm/vfio.c
>> @@ -28,6 +28,7 @@ struct kvm_vfio {
>> struct list_head group_list;
>> struct mutex lock;
>> bool noncoherent;
>> +struct kvm_vfio_arch_data *arch_data;
>> };
>>
>> static struct vfio_group *kvm_vfio_group_get_external_user(struct file
>> *filep)
>> @@ -338,6 +339,26 @@ static int kvm_vfio_create(struct kvm_device *dev, u32
>> type)
>> return 0;
>> }
>>
>> +void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
>> + struct kvm_vfio_arch_data *ptr)
>> +{
>> +kv->arch_data = ptr;
>> +}
>> +
>> +struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio
>> *kv)
>> +{
>
> My preference would be s/get_//
ok
>
>> +return kv->arch_data;
>> +}
>> +
>> +void kvm_vfio_lock(struct kvm_vfio *kv)
>> +{
>> +mutex_lock(&kv->lock);
>> +}
>> +
>> +void kvm_vfio_unlock(struct kvm_vfio *kv)
>> +{
>> +mutex_unlock(&kv->lock);
>> +}
>
> Gosh, what could go wrong...
Hum sorry I did not understand what you meant here
Thanks
Eric
>
>>
>> struct kvm_device_ops kvm_vfio_ops = {
>> .name = "kvm-vfio",
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 5/9] VFIO: Extend external user API
On 08/26/2014 09:02 PM, Alex Williamson wrote:
> On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote:
>> New functions are added to be called from ARM KVM-VFIO device.
>>
>> - vfio_device_get_external_user enables to get a vfio device from
>> its fd
>> - vfio_device_put_external_user puts the vfio device
>> - vfio_external_get_type enables to retrieve the type of the device
>> (PCI or platform)
>> - vfio_external_get_base_device enables to get the
>> struct device*, useful to access the platform_device
>>
>> Signed-off-by: Eric Auger
>> ---
>> drivers/vfio/vfio.c | 35 +++
>> include/linux/vfio.h | 4
>> 2 files changed, 39 insertions(+)
>>
>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>> index 8e84471..c93b9e4 100644
>> --- a/drivers/vfio/vfio.c
>> +++ b/drivers/vfio/vfio.c
>> @@ -1401,6 +1401,41 @@ void vfio_group_put_external_user(struct vfio_group
>> *group)
>> }
>> EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
>>
>> +struct vfio_device *vfio_device_get_external_user(struct file *filep)
>> +{
>> +struct vfio_device *vdev = filep->private_data;
>> +
>> +if (filep->f_op != &vfio_device_fops)
>> +return ERR_PTR(-EINVAL);
>> +
>> +vfio_device_get(vdev);
>> +return vdev;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_device_get_external_user);
>> +
>> +void vfio_device_put_external_user(struct vfio_device *vdev)
>> +{
>> +vfio_device_put(vdev);
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_device_put_external_user);
>> +
>> +int vfio_external_get_type(struct vfio_device *vdev)
>> +{
>> +if (!strcmp(vdev->ops->name, "vfio-platform"))
>> +return VFIO_DEVICE_FLAGS_PLATFORM;
>> +else if (!strcmp(vdev->ops->name, "vfio-pci"))
>> +return VFIO_DEVICE_FLAGS_PCI;
>> +else
>> +return -EINVAL;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_external_get_type);
>
> Returning the bit of the flag we use in get_device_info looks rather
> sloppy here. Should we define a new enum for use with this? Actually,
> is this interface even necessary? If we can get the struct device then
> we can get the bus_type and keep vfio out of this.
thanks for the nit. I will try to get rid of it using it.
>
> For both of these last two, I like to use the convention that where
> there is a "get" there is a matching "put". These aren't reference
> counting anything, so let's not use get in the name.
I will rename.
Thanks
Eric
>
>> +
>> +struct device *vfio_external_get_base_device(struct vfio_device *vdev)
>> +{
>> +return vdev->dev;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_external_get_base_device);
>> +
>
> Looks almost too simple, but reviewing the object lifecycles, this all
> looks safe. Thanks,
>
> Alex
>
>> int vfio_external_user_iommu_id(struct vfio_group *group)
>> {
>> return iommu_group_id(group->iommu_group);
>> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
>> index ffe04ed..19e98eb 100644
>> --- a/include/linux/vfio.h
>> +++ b/include/linux/vfio.h
>> @@ -99,6 +99,10 @@ extern void vfio_group_put_external_user(struct
>> vfio_group *group);
>> extern int vfio_external_user_iommu_id(struct vfio_group *group);
>> extern long vfio_external_check_extension(struct vfio_group *group,
>>unsigned long arg);
>> +extern struct vfio_device *vfio_device_get_external_user(struct file
>> *filep);
>> +extern void vfio_device_put_external_user(struct vfio_device *vdev);
>> +extern int vfio_external_get_type(struct vfio_device *vdev);
>> +extern struct device *vfio_external_get_base_device(struct vfio_device
>> *vdev);
>>
>> struct pci_dev;
>> #ifdef CONFIG_EEH
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 8/9] KVM: KVM-VFIO: add kvm_vfio_arch_data and accessors
On 08/27/2014 05:37 PM, Alex Williamson wrote:
> On Wed, 2014-08-27 at 17:22 +0200, Eric Auger wrote:
>> On 08/26/2014 09:02 PM, Alex Williamson wrote:
>>> On Mon, 2014-08-25 at 15:27 +0200, Eric Auger wrote:
>>>> add a pointer to architecture specific data in kvm_vfio struct
>>>> add accessors to keep kvm_vfio private
>>>>
>>>> Signed-off-by: Eric Auger
>>>> ---
>>>> arch/arm/include/asm/kvm_host.h | 8
>>>> virt/kvm/vfio.c | 21 +
>>>> 2 files changed, 29 insertions(+)
>>>>
>>>> diff --git a/arch/arm/include/asm/kvm_host.h
>>>> b/arch/arm/include/asm/kvm_host.h
>>>> index 62cbf5b..4f1edbf 100644
>>>> --- a/arch/arm/include/asm/kvm_host.h
>>>> +++ b/arch/arm/include/asm/kvm_host.h
>>>> @@ -177,6 +177,14 @@ void kvm_vfio_device_put_external_user(struct
>>>> vfio_device *vdev);
>>>> int kvm_vfio_external_get_type(struct vfio_device *vdev);
>>>> struct device *kvm_vfio_external_get_base_device(struct vfio_device
>>>> *vdev);
>>>>
>>>> +struct kvm_vfio;
>>>> +struct kvm_vfio_arch_data;
>>>> +void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
>>>> + struct kvm_vfio_arch_data *ptr);
>>>> +struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio
>>>> *kv);
>>>> +void kvm_vfio_lock(struct kvm_vfio *kv);
>>>> +void kvm_vfio_unlock(struct kvm_vfio *kv);
>>>> +
>>>> /* We do not have shadow page tables, hence the empty hooks */
>>>> static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
>>>> {
>>>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>>>> index f1c4e35..177b71e 100644
>>>> --- a/virt/kvm/vfio.c
>>>> +++ b/virt/kvm/vfio.c
>>>> @@ -28,6 +28,7 @@ struct kvm_vfio {
>>>>struct list_head group_list;
>>>>struct mutex lock;
>>>>bool noncoherent;
>>>> + struct kvm_vfio_arch_data *arch_data;
>>>> };
>>>>
>>>> static struct vfio_group *kvm_vfio_group_get_external_user(struct file
>>>> *filep)
>>>> @@ -338,6 +339,26 @@ static int kvm_vfio_create(struct kvm_device *dev,
>>>> u32 type)
>>>>return 0;
>>>> }
>>>>
>>>> +void kvm_vfio_device_set_arch_data(struct kvm_vfio *kv,
>>>> + struct kvm_vfio_arch_data *ptr)
>>>> +{
>>>> + kv->arch_data = ptr;
>>>> +}
>>>> +
>>>> +struct kvm_vfio_arch_data *kvm_vfio_device_get_arch_data(struct kvm_vfio
>>>> *kv)
>>>> +{
>>>
>>> My preference would be s/get_//
>> ok
>>>
>>>> + return kv->arch_data;
>>>> +}
>>>> +
>>>> +void kvm_vfio_lock(struct kvm_vfio *kv)
>>>> +{
>>>> + mutex_lock(&kv->lock);
>>>> +}
>>>> +
>>>> +void kvm_vfio_unlock(struct kvm_vfio *kv)
>>>> +{
>>>> + mutex_unlock(&kv->lock);
>>>> +}
>>>
>>> Gosh, what could go wrong...
>> Hum sorry I did not understand what you meant here
>
> Sorry, I was just sarcastically noting that exposing an internal lock
> like this seems to be asking for trouble. As you rework it to pull more
> into the common code and generalize the architecture callouts, I hope we
> can avoid exporting these locks. Thanks,
ok thanks. No problem I learnt a new word ;-)
>
> Alex
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] KVM: EVENTFD: remove inclusion of irq.h
No more needed. irq.h would be void on ARM. Signed-off-by: Eric Auger --- I don't think irq.h is needed anymore since Paul Mackerras' work. However I did not compile for all architectures. --- virt/kvm/eventfd.c | 1 - 1 file changed, 1 deletion(-) diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 3c5981c..0c712a7 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -36,7 +36,6 @@ #include #include -#include "irq.h" #include "iodev.h" #ifdef CONFIG_HAVE_KVM_IRQFD -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3] ARM: KVM: add irqfd support
This patch enables irqfd on ARM.
irqfd framework enables to inject a virtual IRQ into a guest upon an
eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
a kvm_irqfd struct that associates a VM, an eventfd, a virtual IRQ number
(aka. the gsi). When an actor signals the eventfd (typically a VFIO
platform driver), the kvm irqfd subsystem injects the provided virtual
IRQ into the guest.
Resamplefd also is supported for level sensitive interrupts, ie. the
user can provide another eventfd that is triggered when the completion
of the virtual IRQ (gsi) is detected by the GIC.
The gsi must correspond to a shared peripheral interrupt (SPI), ie the
GIC interrupt ID is gsi+32.
this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
CONFIG_HAVE_KVM_IRQCHIP is removed. No IRQ routing table is used
(irqchip.c and irqcomm.c are not used).
Both KVM_CAP_IRQFD & KVM_CAP_IRQFD_RESAMPLE capabilities are exposed
Signed-off-by: Eric Auger
---
This patch serie deprecates the previous serie featuring GSI routing
(https://patches.linaro.org/32261/)
The patch serie has the following dependencies:
- arm/arm64: KVM: Various VGIC cleanups and improvements
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
- "KVM: EVENTFD: remove inclusion of irq.h"
All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
branch irqfd_norouting_integ_v3
This work was tested with Calxeda Midway xgmac main interrupt with
qemu-system-arm and QEMU VFIO platform device.
v2 -> v3:
- removal of irq.h from eventfd.c put in a separate patch to increase
visibility
- properly expose KVM_CAP_IRQFD capability in arm.c
- remove CONFIG_HAVE_KVM_IRQCHIP meaningfull only if irq_comm.c is used
v1 -> v2:
- rebase on 3.17rc1
- move of the dist unlock in process_maintenance
- remove of dist lock in __kvm_vgic_sync_hwstate
- rewording of the commit message (add resamplefd reference)
- remove irq.h
---
Documentation/virtual/kvm/api.txt | 5 +++-
arch/arm/include/uapi/asm/kvm.h | 3 +++
arch/arm/kvm/Kconfig | 4 +--
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/arm.c| 3 +++
virt/kvm/arm/vgic.c | 56 ---
6 files changed, 65 insertions(+), 8 deletions(-)
diff --git a/Documentation/virtual/kvm/api.txt
b/Documentation/virtual/kvm/api.txt
index beae3fd..8118b12 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2204,7 +2204,7 @@ into the hash PTE second double word).
4.75 KVM_IRQFD
Capability: KVM_CAP_IRQFD
-Architectures: x86 s390
+Architectures: x86 s390 arm
Type: vm ioctl
Parameters: struct kvm_irqfd (in)
Returns: 0 on success, -1 on error
@@ -2230,6 +2230,9 @@ Note that closing the resamplefd is not sufficient to
disable the
irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
+On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
+This means the programmed GIC interrupt ID is gsi+32.
+
4.76 KVM_PPC_ALLOCATE_HTAB
Capability: KVM_CAP_PPC_ALLOC_HTAB
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index e6ebdd3..3034c66 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
/* Highest supported SPI, from VGIC_NR_IRQS */
#define KVM_ARM_IRQ_GIC_MAX127
+/* One single KVM irqchip, ie. the VGIC */
+#define KVM_NR_IRQCHIPS 1
+
/* PSCI interface */
#define KVM_PSCI_FN_BASE 0x95c1ba5e
#define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 466bd29..e519a40 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -24,6 +24,7 @@ config KVM
select KVM_MMIO
select KVM_ARM_HOST
depends on ARM_VIRT_EXT && ARM_LPAE
+ select HAVE_KVM_EVENTFD
---help---
Support hosting virtualized guest machines. You will also
need to select one or more of the processor modules below.
@@ -55,7 +56,7 @@ config KVM_ARM_MAX_VCPUS
config KVM_ARM_VGIC
bool "KVM support for Virtual GIC"
depends on KVM_ARM_HOST && OF
- select HAVE_KVM_IRQCHIP
+ select HAVE_KVM_IRQFD
default y
---help---
Adds support for a hardware assisted, in-kernel GIC emulation.
@@ -63,7 +64,6 @@ config KVM_ARM_VGIC
config KVM_ARM_TIMER
bool "KVM support for Architected Timers"
depends on KVM_ARM_VGIC && ARM_ARCH_TIMER
- select HAVE_KVM_IRQCHIP
default y
---help---
Adds support for the Architected Timers in virtual machines
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index f7057ed..859db09 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
[RFC v2 9/9] KVM: KVM-VFIO: ARM forwarding control
Enables forwarding control for ARM. By defining
__KVM_HAVE_ARCH_KVM_VFIO_FORWARD the patch enables
KVM_DEV_VFIO_DEVICE_FORWARD/UNFORWARD_IRQ command on ARM. As a
result it brings an optimized injection/completion handling for
forwarded IRQ. The ARM specific part is implemented in a new module,
kvm_vfio_arm.c
Signed-off-by: Eric Auger
---
arch/arm/include/asm/kvm_host.h | 2 +
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/kvm_vfio_arm.c | 85 +
3 files changed, 88 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/kvm/kvm_vfio_arm.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 1aee6bb..dfd3b05 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -25,6 +25,8 @@
#include
#include
+#define __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
+
#if defined(CONFIG_KVM_ARM_MAX_VCPUS)
#define KVM_MAX_VCPUS CONFIG_KVM_ARM_MAX_VCPUS
#else
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index ea1fa76..26a5a42 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -19,7 +19,7 @@ kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
$(KVM)/eventfd.o $(KVM)/vf
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
-obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o
+obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o kvm_vfio_arm.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic-v2.o
obj-$(CONFIG_KVM_ARM_TIMER) += $(KVM)/arm/arch_timer.o
diff --git a/arch/arm/kvm/kvm_vfio_arm.c b/arch/arm/kvm/kvm_vfio_arm.c
new file mode 100644
index 000..0d316b1
--- /dev/null
+++ b/arch/arm/kvm/kvm_vfio_arm.c
@@ -0,0 +1,85 @@
+/*
+ * Copyright (C) 2014 Linaro Ltd.
+ * Authors: Eric Auger
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+/**
+ * kvm_arch_set_fwd_state - change the forwarded state of an IRQ
+ * @pfwd: the forwarded irq struct
+ * @action: action to perform (set forward, set back normal, cleanup)
+ *
+ * programs the GIC and VGIC
+ * returns the VGIC map/unmap return status
+ * It is the responsability of the caller to make sure the physical IRQ
+ * is not active. there is a critical section between the start of the
+ * VFIO IRQ handler and LR programming.
+ */
+int kvm_arch_set_fwd_state(struct kvm_fwd_irq *pfwd,
+ enum kvm_fwd_irq_action action)
+{
+ int ret;
+ struct irq_desc *desc = irq_to_desc(pfwd->hwirq);
+ struct irq_data *d = &desc->irq_data;
+ struct irq_chip *chip = desc->irq_data.chip;
+
+ disable_irq(pfwd->hwirq);
+ /* no fwd state change can happen if the IRQ is in progress */
+ if (irqd_irq_inprogress(d)) {
+ kvm_err("%s cannot change fwd state (IRQ %d in progress\n",
+ __func__, pfwd->hwirq);
+ enable_irq(pfwd->hwirq);
+ return -1;
+ }
+
+ if (action == KVM_VFIO_IRQ_SET_FORWARD) {
+ irqd_set_irq_forwarded(d);
+ ret = vgic_map_phys_irq(pfwd->vcpu,
+ pfwd->gsi + VGIC_NR_PRIVATE_IRQS,
+ pfwd->hwirq);
+ } else if (action == KVM_VFIO_IRQ_SET_NORMAL) {
+ irqd_clr_irq_forwarded(d);
+ ret = vgic_unmap_phys_irq(pfwd->vcpu,
+ pfwd->gsi +
+ VGIC_NR_PRIVATE_IRQS,
+ pfwd->hwirq);
+ } else if (action == KVM_VFIO_IRQ_CLEANUP) {
+ irqd_clr_irq_forwarded(d);
+ /*
+* in case the guest did not complete the
+* virtual IRQ, let's do it for him.
+* when cleanup is called, VCPU have already
+* been freed, do not manipulate VGIC
+*/
+ chip->irq_eoi(d);
+ ret = 0;
+ } else {
+ enable_irq(pfwd->hwirq);
+ ret = -EINVAL;
+ }
+
+ enable_irq(pfwd->hwirq);
+ return ret;
+}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 8/9] KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ forwarding control
This patch introduces a new KVM_DEV_VFIO_DEVICE attribute.
This is a new control channel which enables KVM to cooperate with
viable VFIO devices.
The kvm-vfio device now holds a list of devices (kvm_vfio_device)
in addition to a list of groups (kvm_vfio_group). The new
infrastructure enables to check the validity of the VFIO device
file descriptor, get and hold a reference to it.
The first concrete implemented command is IRQ forward control:
KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
It consists in programing the VFIO driver and KVM in a consistent manner
so that an optimized IRQ injection/completion is set up. Each
kvm_vfio_device holds a list of forwarded IRQ. When putting a
kvm_vfio_device, the implementation makes sure the forwarded IRQs
are set again in the normal handling state (non forwarded).
The forwarding programmming is architecture specific, embodied by the
kvm_arch_set_fwd_state function. Its implementation is given in a
separate patch file.
The forwarding control modality is enabled by the
__KVM_HAVE_ARCH_KVM_VFIO_FORWARD define.
Signed-off-by: Eric Auger
---
v1 -> v2:
- __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
- original patch file separated into 2 parts: generic part moved in vfio.c
and ARM specific part(kvm_arch_set_fwd_state)
---
include/linux/kvm_host.h | 27 +++
virt/kvm/vfio.c | 452 ++-
2 files changed, 477 insertions(+), 2 deletions(-)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index a4c33b3..24350dc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1065,6 +1065,21 @@ struct kvm_device_ops {
unsigned long arg);
};
+enum kvm_fwd_irq_action {
+ KVM_VFIO_IRQ_SET_FORWARD,
+ KVM_VFIO_IRQ_SET_NORMAL,
+ KVM_VFIO_IRQ_CLEANUP,
+};
+
+/* internal structure describing a forwarded IRQ */
+struct kvm_fwd_irq {
+ struct list_head link;
+ __u32 index; /* platform device irq index */
+ __u32 hwirq; /*physical IRQ */
+ __u32 gsi; /* virtual IRQ */
+ struct kvm_vcpu *vcpu; /* vcpu to inject into*/
+};
+
void kvm_device_get(struct kvm_device *dev);
void kvm_device_put(struct kvm_device *dev);
struct kvm_device *kvm_device_from_filp(struct file *filp);
@@ -1075,6 +1090,18 @@ extern struct kvm_device_ops kvm_vfio_ops;
extern struct kvm_device_ops kvm_arm_vgic_v2_ops;
extern struct kvm_device_ops kvm_flic_ops;
+#ifdef __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
+int kvm_arch_set_fwd_state(struct kvm_fwd_irq *pfwd,
+ enum kvm_fwd_irq_action action);
+
+#else
+static inline int kvm_arch_set_fwd_state(struct kvm_fwd_irq *pfwd,
+enum kvm_fwd_irq_action action)
+{
+ return 0;
+}
+#endif
+
#ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
static inline void kvm_vcpu_set_in_spin_loop(struct kvm_vcpu *vcpu, bool val)
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index 76dc7a1..e4a81c4 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -18,14 +18,24 @@
#include
#include
#include
+#include
struct kvm_vfio_group {
struct list_head node;
struct vfio_group *vfio_group;
};
+struct kvm_vfio_device {
+ struct list_head node;
+ struct vfio_device *vfio_device;
+ /* list of forwarded IRQs for that VFIO device */
+ struct list_head fwd_irq_list;
+ int fd;
+};
+
struct kvm_vfio {
struct list_head group_list;
+ struct list_head device_list;
struct mutex lock;
bool noncoherent;
};
@@ -246,12 +256,441 @@ static int kvm_vfio_set_group(struct kvm_device *dev,
long attr, u64 arg)
return -ENXIO;
}
+/**
+ * get_vfio_device - returns the vfio-device corresponding to this fd
+ * @fd:fd of the vfio platform device
+ *
+ * checks it is a vfio device
+ * increment its ref counter
+ */
+static struct vfio_device *kvm_vfio_get_vfio_device(int fd)
+{
+ struct fd f;
+ struct vfio_device *vdev;
+
+ f = fdget(fd);
+ if (!f.file)
+ return NULL;
+ vdev = kvm_vfio_device_get_external_user(f.file);
+ fdput(f);
+ return vdev;
+}
+
+/**
+ * put_vfio_device: put the vfio platform device
+ * @vdev: vfio_device to put
+ *
+ * decrement the ref counter
+ */
+static void kvm_vfio_put_vfio_device(struct vfio_device *vdev)
+{
+ kvm_vfio_device_put_external_user(vdev);
+}
+
+/**
+ * kvm_vfio_find_device - look for the device in the assigned
+ * device list
+ * @kv: the kvm-vfio device
+ * @vdev: the vfio_device to look for
+ *
+ * returns the associated kvm_vfio_device if the device is known,
+ * meaning at least 1 IRQ is forwarded for this device.
+ * in the device is not registered, returns NULL.
+ */
+struct kvm_vfio_device *kvm_vfio_find_device(struct kvm_vfio *kv,
+struct vfio_device *vdev)
+{
+ struct kvm_vfio_device *kvm_vdev_i
[RFC v2 7/9] KVM: KVM-VFIO: add new VFIO external API hooks
add functions that implement the gateway to the extended
external VFIO API:
- kvm_vfio_device_get_external_user
- kvm_vfio_device_put_external_user
- kvm_vfio_external_base_device
Signed-off-by: Eric Auger
---
v1 -> v2:
- kvm_vfio_external_get_base_device renamed into
kvm_vfio_external_base_device
- kvm_vfio_external_get_type removed
---
arch/arm/include/asm/kvm_host.h | 5 +
virt/kvm/vfio.c | 45 +
2 files changed, 50 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 6dfb404..1aee6bb 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -171,6 +171,11 @@ void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva,
pte_t pte);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
+struct vfio_device;
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep);
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev);
+struct device *kvm_vfio_external_base_device(struct vfio_device *vdev);
+
/* We do not have shadow page tables, hence the empty hooks */
static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
{
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index ba1a93f..76dc7a1 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -59,6 +59,51 @@ static void kvm_vfio_group_put_external_user(struct
vfio_group *vfio_group)
symbol_put(vfio_group_put_external_user);
}
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev;
+ struct vfio_device *(*fn)(struct file *);
+
+ fn = symbol_get(vfio_device_get_external_user);
+ if (!fn)
+ return ERR_PTR(-EINVAL);
+
+ vdev = fn(filep);
+
+ symbol_put(vfio_device_get_external_user);
+
+ return vdev;
+}
+
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ void (*fn)(struct vfio_device *);
+
+ fn = symbol_get(vfio_device_put_external_user);
+ if (!fn)
+ return;
+
+ fn(vdev);
+
+ symbol_put(vfio_device_put_external_user);
+}
+
+struct device *kvm_vfio_external_base_device(struct vfio_device *vdev)
+{
+ struct device *(*fn)(struct vfio_device *);
+ struct device *dev;
+
+ fn = symbol_get(vfio_external_base_device);
+ if (!fn)
+ return NULL;
+
+ dev = fn(vdev);
+
+ symbol_put(vfio_external_base_device);
+
+ return dev;
+}
+
static bool kvm_vfio_group_is_coherent(struct vfio_group *vfio_group)
{
long (*fn)(struct vfio_group *, unsigned long);
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 1/9] KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded IRQ
Fix multiple injection of level sensitive forwarded IRQs.
With current code, the second injection fails since the state bitmaps
are not reset (process_maintenance is not called anymore).
New implementation consists in fully bypassing the vgic state
management for forwarded IRQ (checks are ignored in
vgic_update_irq_pending). This obviously assumes the forwarded IRQ is
injected from kernel side.
Signed-off-by: Eric Auger
---
It was attempted to reset the states in __kvm_vgic_sync_hwstate, checking
the emptied LR of forwarded IRQ. However surprisingly this solution does
not seem to work. Some times, a new forwarded IRQ injection is observed
while the LR of the previous instance was not observed as empty.
v1 -> v2:
- fix vgic state bypass in vgic_queue_hwirq
---
virt/kvm/arm/vgic.c | 13 ++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 0007300..8ef495b 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1259,7 +1259,9 @@ static bool vgic_queue_sgi(struct kvm_vcpu *vcpu, int irq)
static bool vgic_queue_hwirq(struct kvm_vcpu *vcpu, int irq)
{
- if (vgic_irq_is_queued(vcpu, irq))
+ bool is_forwarded = (vgic_get_phys_irq(vcpu, irq) > 0);
+
+ if (vgic_irq_is_queued(vcpu, irq) && !is_forwarded)
return true; /* level interrupt, already queued */
if (vgic_queue_irq(vcpu, 0, irq)) {
@@ -1517,14 +1519,18 @@ static bool vgic_update_irq_pending(struct kvm *kvm,
int cpuid,
int edge_triggered, level_triggered;
int enabled;
bool ret = true;
+ bool is_forwarded;
spin_lock(&dist->lock);
vcpu = kvm_get_vcpu(kvm, cpuid);
+ is_forwarded = (vgic_get_phys_irq(vcpu, irq_num) > 0);
+
edge_triggered = vgic_irq_is_edge(vcpu, irq_num);
level_triggered = !edge_triggered;
- if (!vgic_validate_injection(vcpu, irq_num, level)) {
+ if (!is_forwarded &&
+ !vgic_validate_injection(vcpu, irq_num, level)) {
ret = false;
goto out;
}
@@ -1557,7 +1563,8 @@ static bool vgic_update_irq_pending(struct kvm *kvm, int
cpuid,
goto out;
}
- if (level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
+ if (!is_forwarded &&
+ level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
/*
* Level interrupt in progress, will be picked up
* when EOId.
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 3/9] ARM: KVM: Enable the KVM-VFIO device
From: Kim Phillips Used by KVM-enabled VFIO-based device passthrough support in QEMU. Signed-off-by: Kim Phillips --- arch/arm/kvm/Kconfig | 1 + arch/arm/kvm/Makefile | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig index e519a40..aace254 100644 --- a/arch/arm/kvm/Kconfig +++ b/arch/arm/kvm/Kconfig @@ -24,6 +24,7 @@ config KVM select KVM_MMIO select KVM_ARM_HOST depends on ARM_VIRT_EXT && ARM_LPAE + select KVM_VFIO select HAVE_KVM_EVENTFD ---help--- Support hosting virtualized guest machines. You will also diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile index 859db09..ea1fa76 100644 --- a/arch/arm/kvm/Makefile +++ b/arch/arm/kvm/Makefile @@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt) AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt) KVM := ../../../virt/kvm -kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o +kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o obj-y += kvm-arm.o init.o interrupts.o obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 2/9] KVM: ARM: VGIC: add forwarded irq rbtree lock
add a lock related to the rb tree manipulation. The rb tree can be
searched in one thread (irqfd handler for instance) and map/unmap
happen in another.
Signed-off-by: Eric Auger
---
include/kvm/arm_vgic.h | 1 +
virt/kvm/arm/vgic.c| 46 +-
2 files changed, 38 insertions(+), 9 deletions(-)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 743020f..3da244f 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -177,6 +177,7 @@ struct vgic_dist {
unsigned long irq_pending_on_cpu;
struct rb_root irq_phys_map;
+ spinlock_t rb_tree_lock;
#endif
};
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 8ef495b..dbc2a5a 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1630,9 +1630,15 @@ static struct rb_root *vgic_get_irq_phys_map(struct
kvm_vcpu *vcpu,
int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct rb_root *root = vgic_get_irq_phys_map(vcpu, virt_irq);
- struct rb_node **new = &root->rb_node, *parent = NULL;
+ struct rb_root *root;
+ struct rb_node **new, *parent = NULL;
struct irq_phys_map *new_map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ spin_lock(&dist->rb_tree_lock);
+
+ root = vgic_get_irq_phys_map(vcpu, virt_irq);
+ new = &root->rb_node;
/* Boilerplate rb_tree code */
while (*new) {
@@ -1644,13 +1650,17 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
virt_irq, int phys_irq)
new = &(*new)->rb_left;
else if (this->virt_irq > virt_irq)
new = &(*new)->rb_right;
- else
+ else {
+ spin_unlock(&dist->rb_tree_lock);
return -EEXIST;
+ }
}
new_map = kzalloc(sizeof(*new_map), GFP_KERNEL);
- if (!new_map)
+ if (!new_map) {
+ spin_unlock(&dist->rb_tree_lock);
return -ENOMEM;
+ }
new_map->virt_irq = virt_irq;
new_map->phys_irq = phys_irq;
@@ -1658,6 +1668,8 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
virt_irq, int phys_irq)
rb_link_node(&new_map->node, parent, new);
rb_insert_color(&new_map->node, root);
+ spin_unlock(&dist->rb_tree_lock);
+
return 0;
}
@@ -1685,24 +1697,39 @@ static struct irq_phys_map *vgic_irq_map_search(struct
kvm_vcpu *vcpu,
int vgic_get_phys_irq(struct kvm_vcpu *vcpu, int virt_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+ int ret;
+
+ spin_lock(&dist->rb_tree_lock);
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map)
- return map->phys_irq;
+ ret = map->phys_irq;
+ else
+ ret = -ENOENT;
+
+ spin_unlock(&dist->rb_tree_lock);
+ return ret;
- return -ENOENT;
}
int vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ spin_lock(&dist->rb_tree_lock);
+
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map && map->phys_irq == phys_irq) {
rb_erase(&map->node, vgic_get_irq_phys_map(vcpu, virt_irq));
kfree(map);
+ spin_unlock(&dist->rb_tree_lock);
return 0;
}
-
+ spin_unlock(&dist->rb_tree_lock);
return -ENOENT;
}
@@ -1898,6 +1925,7 @@ int kvm_vgic_create(struct kvm *kvm)
}
spin_lock_init(&kvm->arch.vgic.lock);
+ spin_lock_init(&kvm->arch.vgic.rb_tree_lock);
kvm->arch.vgic.in_kernel = true;
kvm->arch.vgic.vctrl_base = vgic->vctrl_base;
kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 6/9] VFIO: Extend external user API
New functions are added to be called from ARM KVM-VFIO device.
- vfio_device_get_external_user enables to get a vfio device from
its fd
- vfio_device_put_external_user puts the vfio device
- vfio_external_base_device returns the struct device*,
useful to access the platform_device
Signed-off-by: Eric Auger
---
v1 -> v2:
- vfio_external_get_base_device renamed into vfio_external_base_device
- vfio_external_get_type removed
---
drivers/vfio/vfio.c | 24
include/linux/vfio.h | 3 +++
2 files changed, 27 insertions(+)
diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index 8e84471..282814e 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1401,6 +1401,30 @@ void vfio_group_put_external_user(struct vfio_group
*group)
}
EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
+struct vfio_device *vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev = filep->private_data;
+
+ if (filep->f_op != &vfio_device_fops)
+ return ERR_PTR(-EINVAL);
+
+ vfio_device_get(vdev);
+ return vdev;
+}
+EXPORT_SYMBOL_GPL(vfio_device_get_external_user);
+
+void vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ vfio_device_put(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_device_put_external_user);
+
+struct device *vfio_external_base_device(struct vfio_device *vdev)
+{
+ return vdev->dev;
+}
+EXPORT_SYMBOL_GPL(vfio_external_base_device);
+
int vfio_external_user_iommu_id(struct vfio_group *group)
{
return iommu_group_id(group->iommu_group);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index ffe04ed..bd4b6cb 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -99,6 +99,9 @@ extern void vfio_group_put_external_user(struct vfio_group
*group);
extern int vfio_external_user_iommu_id(struct vfio_group *group);
extern long vfio_external_check_extension(struct vfio_group *group,
unsigned long arg);
+extern struct vfio_device *vfio_device_get_external_user(struct file *filep);
+extern void vfio_device_put_external_user(struct vfio_device *vdev);
+extern struct device *vfio_external_base_device(struct vfio_device *vdev);
struct pci_dev;
#ifdef CONFIG_EEH
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 5/9] KVM: KVM-VFIO: update user API to program forwarded IRQ
add new device group commands:
- KVM_DEV_VFIO_DEVICE_FORWARD_IRQ and
KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ
which enable to turn forwarded IRQ mode on/off.
the kvm_arch_forwarded_irq struct embodies a forwarded IRQ
Signed-off-by: Eric Auger
---
v1 -> v2:
- struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h
to include/uapi/linux/kvm.h
also irq_index renamed into index and guest_irq renamed into gsi
- ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD
---
Documentation/virtual/kvm/devices/vfio.txt | 26 ++
include/uapi/linux/kvm.h | 9 +
2 files changed, 35 insertions(+)
diff --git a/Documentation/virtual/kvm/devices/vfio.txt
b/Documentation/virtual/kvm/devices/vfio.txt
index ef51740..048baa0 100644
--- a/Documentation/virtual/kvm/devices/vfio.txt
+++ b/Documentation/virtual/kvm/devices/vfio.txt
@@ -13,6 +13,7 @@ VFIO-group is held by KVM.
Groups:
KVM_DEV_VFIO_GROUP
+ KVM_DEV_VFIO_DEVICE
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
@@ -20,3 +21,28 @@ KVM_DEV_VFIO_GROUP attributes:
For each, kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
+
+KVM_DEV_VFIO_DEVICE attributes:
+ KVM_DEV_VFIO_DEVICE_FORWARD_IRQ
+ KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ
+
+For each, kvm_device_attr.addr points to a kvm_arch_forwarded_irq struct.
+This user API makes possible to create a special IRQ handling mode,
+where KVM and a VFIO platform driver collaborate to improve IRQ
+handling performance.
+
+fd represents the file descriptor of a valid VFIO device whose physical
+IRQ, referenced by its index, is injected into the VM guest irq (gsi).
+
+On FORWARD_IRQ, KVM-VFIO device programs:
+- the host, to not complete the physical IRQ itself.
+- the GIC, to automatically complete the physical IRQ when the guest
+ completes the virtual IRQ.
+This avoids trapping the end-of-interrupt for level sensitive IRQ.
+
+On UNFORWARD_IRQ, one returns to the mode where the host completes the
+physical IRQ and the guest completes the virtual IRQ.
+
+It is up to the caller of this API to make sure the IRQ is not
+outstanding when the FORWARD/UNFORWARD is called. This could lead to
+some inconsistency on who is going to complete the IRQ.
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index cf3a2ff..8cd7b0e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -947,6 +947,12 @@ struct kvm_device_attr {
__u64 addr; /* userspace address of attr data */
};
+struct kvm_arch_forwarded_irq {
+ __u32 fd; /* file desciptor of the VFIO device */
+ __u32 index; /* VFIO device IRQ index */
+ __u32 gsi; /* gsi, ie. virtual IRQ number */
+};
+
#define KVM_DEV_TYPE_FSL_MPIC_20 1
#define KVM_DEV_TYPE_FSL_MPIC_42 2
#define KVM_DEV_TYPE_XICS 3
@@ -954,6 +960,9 @@ struct kvm_device_attr {
#define KVM_DEV_VFIO_GROUP1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_DEVICE 2
+#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
+#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
#define KVM_DEV_TYPE_ARM_VGIC_V2 5
#define KVM_DEV_TYPE_FLIC 6
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 0/9] KVM-VFIO IRQ forward control
This RFC proposes an integration of "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) in KVM. It enables to transform a VFIO platform driver IRQ into a forwarded IRQ. The direct benefit is that, for a level sensitive IRQ, a VM switch can be avoided on guest virtual IRQ completion. Before this patch, a maintenance IRQ was triggered on the virtual IRQ completion. When the IRQ is forwarded, the VFIO platform driver does not need to disable the IRQ anymore. Indeed when returning from the IRQ handler the IRQ is not deactivated. Only its priority is lowered. This means the same IRQ cannot hit before the guest completes the virtual IRQ and the GIC automatically deactivates the corresponding physical IRQ. Besides, the injection still is based on irqfd triggering. The only impact on irqfd process is resamplefd is not called anymore on virtual IRQ completion since this latter becomes "transparent". The current integration is based on an extension of the KVM-VFIO device, previously used by KVM to interact with VFIO groups. The patch serie now enables KVM to directly interact with a VFIO platform device. The VFIO external API was extended for that purpose. Th KVM-VFIO device can get/put the vfio platform device, check its integrity and type, get the IRQ number associated to an IRQ index. The IRQ forward programming is architecture specific (virtual interrupt controller programming basically). However the whole infrastructure is kept generic. from a user point of view, the functionality is provided through new KVM-VFIO device commands, KVM_DEV_VFIO_DEVICE_(UN)FORWARD_IRQ and the capability can be checked with KVM_HAS_DEVICE_ATTR. Assignment can only be changed when the physical IRQ is not active. It is the responsability of the user to do this check. This patch serie has the following dependencies: - "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) in - [PATCH v3] irqfd for ARM - and obviously the VFIO platform driver serie: [RFC PATCH v6 00/20] VFIO support for platform devices on ARM https://www.mail-archive.com/kvm@vger.kernel.org/msg103247.html Integrated pieces can be found at ssh://git.linaro.org/people/eric.auger/linux.git on branch 3.17rc3_irqfd_forward_integ_v2 This was was tested on Calxeda Midway, assigning the xgmac main IRQ. v1 -> v2: - forward control is moved from architecture specific file into generic vfio.c module. only kvm_arch_set_fwd_state remains architecture specific - integrate Kim's patch which enables KVM-VFIO for ARM - fix vgic state bypass in vgic_queue_hwirq - struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h to include/uapi/linux/kvm.h also irq_index renamed into index and guest_irq renamed into gsi - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD - vfio_external_get_base_device renamed into vfio_external_base_device - vfio_external_get_type removed - kvm_vfio_external_get_base_device renamed into kvm_vfio_external_base_device - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD Eric Auger (8): KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded IRQ KVM: ARM: VGIC: add forwarded irq rbtree lock VFIO: platform: handler tests whether the IRQ is forwarded KVM: KVM-VFIO: update user API to program forwarded IRQ VFIO: Extend external user API KVM: KVM-VFIO: add new VFIO external API hooks KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ forwarding control KVM: KVM-VFIO: ARM forwarding control Kim Phillips (1): ARM: KVM: Enable the KVM-VFIO device Documentation/virtual/kvm/devices/vfio.txt | 26 ++ arch/arm/include/asm/kvm_host.h| 7 + arch/arm/kvm/Kconfig | 1 + arch/arm/kvm/Makefile | 4 +- arch/arm/kvm/kvm_vfio_arm.c| 85 + drivers/vfio/platform/vfio_platform_irq.c | 7 +- drivers/vfio/vfio.c| 24 ++ include/kvm/arm_vgic.h | 1 + include/linux/kvm_host.h | 27 ++ include/linux/vfio.h | 3 + include/uapi/linux/kvm.h | 9 + virt/kvm/arm/vgic.c| 59 +++- virt/kvm/vfio.c| 497 - 13 files changed, 733 insertions(+), 17 deletions(-) create mode 100644 arch/arm/kvm/kvm_vfio_arm.c -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2 4/9] VFIO: platform: handler tests whether the IRQ is forwarded
In case the IRQ is forwarded, the VFIO platform IRQ handler does not
need to disable the IRQ anymore. In that mode, when the handler completes
the IRQ is not deactivated but only its priority is lowered.
Some other actor (typically a guest) is supposed to deactivate the IRQ,
allowing at that time a new physical IRQ to hit.
In virtualization use case, the physical IRQ is automatically completed
by the interrupt controller when the guest completes the corresponding
virtual IRQ.
Signed-off-by: Eric Auger
---
drivers/vfio/platform/vfio_platform_irq.c | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vfio/platform/vfio_platform_irq.c
b/drivers/vfio/platform/vfio_platform_irq.c
index 6768508..1f851b2 100644
--- a/drivers/vfio/platform/vfio_platform_irq.c
+++ b/drivers/vfio/platform/vfio_platform_irq.c
@@ -88,13 +88,18 @@ static irqreturn_t vfio_irq_handler(int irq, void *dev_id)
struct vfio_platform_irq *irq_ctx = dev_id;
unsigned long flags;
int ret = IRQ_NONE;
+ struct irq_data *d;
+ bool is_forwarded;
spin_lock_irqsave(&irq_ctx->lock, flags);
if (!irq_ctx->masked) {
ret = IRQ_HANDLED;
+ d = irq_get_irq_data(irq_ctx->hwirq);
+ is_forwarded = irqd_irq_forwarded(d);
- if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED) {
+ if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED &&
+ !is_forwarded) {
disable_irq_nosync(irq_ctx->hwirq);
irq_ctx->masked = true;
}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2 0/9] KVM-VFIO IRQ forward control
On 09/02/2014 11:05 PM, Alex Williamson wrote: > On Mon, 2014-09-01 at 14:52 +0200, Eric Auger wrote: >> This RFC proposes an integration of "ARM: Forwarding physical >> interrupts to a guest VM" (http://lwn.net/Articles/603514/) in >> KVM. >> >> It enables to transform a VFIO platform driver IRQ into a forwarded >> IRQ. The direct benefit is that, for a level sensitive IRQ, a VM >> switch can be avoided on guest virtual IRQ completion. Before this >> patch, a maintenance IRQ was triggered on the virtual IRQ completion. >> >> When the IRQ is forwarded, the VFIO platform driver does not need to >> disable the IRQ anymore. Indeed when returning from the IRQ handler >> the IRQ is not deactivated. Only its priority is lowered. This means >> the same IRQ cannot hit before the guest completes the virtual IRQ >> and the GIC automatically deactivates the corresponding physical IRQ. >> >> Besides, the injection still is based on irqfd triggering. The only >> impact on irqfd process is resamplefd is not called anymore on >> virtual IRQ completion since this latter becomes "transparent". >> >> The current integration is based on an extension of the KVM-VFIO >> device, previously used by KVM to interact with VFIO groups. The >> patch serie now enables KVM to directly interact with a VFIO >> platform device. The VFIO external API was extended for that purpose. >> >> Th KVM-VFIO device can get/put the vfio platform device, check its >> integrity and type, get the IRQ number associated to an IRQ index. >> >> The IRQ forward programming is architecture specific (virtual interrupt >> controller programming basically). However the whole infrastructure is >> kept generic. >> >> from a user point of view, the functionality is provided through new >> KVM-VFIO device commands, KVM_DEV_VFIO_DEVICE_(UN)FORWARD_IRQ >> and the capability can be checked with KVM_HAS_DEVICE_ATTR. >> Assignment can only be changed when the physical IRQ is not active. >> It is the responsability of the user to do this check. >> >> This patch serie has the following dependencies: >> - "ARM: Forwarding physical interrupts to a guest VM" >> (http://lwn.net/Articles/603514/) in >> - [PATCH v3] irqfd for ARM >> - and obviously the VFIO platform driver serie: >> [RFC PATCH v6 00/20] VFIO support for platform devices on ARM >> https://www.mail-archive.com/kvm@vger.kernel.org/msg103247.html >> >> Integrated pieces can be found at >> ssh://git.linaro.org/people/eric.auger/linux.git >> on branch 3.17rc3_irqfd_forward_integ_v2 >> >> This was was tested on Calxeda Midway, assigning the xgmac main IRQ. >> >> v1 -> v2: >> - forward control is moved from architecture specific file into generic >> vfio.c module. >> only kvm_arch_set_fwd_state remains architecture specific >> - integrate Kim's patch which enables KVM-VFIO for ARM >> - fix vgic state bypass in vgic_queue_hwirq >> - struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h >> to include/uapi/linux/kvm.h >> also irq_index renamed into index and guest_irq renamed into gsi >> - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD >> - vfio_external_get_base_device renamed into vfio_external_base_device >> - vfio_external_get_type removed >> - kvm_vfio_external_get_base_device renamed into >> kvm_vfio_external_base_device >> - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD >> >> Eric Auger (8): >> KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded >> IRQ >> KVM: ARM: VGIC: add forwarded irq rbtree lock >> VFIO: platform: handler tests whether the IRQ is forwarded >> KVM: KVM-VFIO: update user API to program forwarded IRQ >> VFIO: Extend external user API >> KVM: KVM-VFIO: add new VFIO external API hooks >> KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ forwarding >> control >> KVM: KVM-VFIO: ARM forwarding control >> >> Kim Phillips (1): >> ARM: KVM: Enable the KVM-VFIO device >> >> Documentation/virtual/kvm/devices/vfio.txt | 26 ++ >> arch/arm/include/asm/kvm_host.h| 7 + >> arch/arm/kvm/Kconfig | 1 + >> arch/arm/kvm/Makefile | 4 +- >> arch/arm/kvm/kvm_vfio_arm.c| 85 + >> drivers/vfio/platform/vfio_platform_irq.c | 7 +- >> drivers/vfio/vfio.c| 24 ++ >> include/kvm/arm_vgic.h | 1 + >&
Re: [PATCH v3] ARM: KVM: add irqfd support
On 09/11/2014 05:09 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 10:53:04AM +0200, Eric Auger wrote:
>> This patch enables irqfd on ARM.
>>
>> irqfd framework enables to inject a virtual IRQ into a guest upon an
>> eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with
>> a kvm_irqfd struct that associates a VM, an eventfd, a virtual IRQ number
>> (aka. the gsi). When an actor signals the eventfd (typically a VFIO
>> platform driver), the kvm irqfd subsystem injects the provided virtual
>> IRQ into the guest.
>>
>> Resamplefd also is supported for level sensitive interrupts, ie. the
>> user can provide another eventfd that is triggered when the completion
>> of the virtual IRQ (gsi) is detected by the GIC.
>>
>> The gsi must correspond to a shared peripheral interrupt (SPI), ie the
>> GIC interrupt ID is gsi+32.
>>
>> this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
>> CONFIG_HAVE_KVM_IRQCHIP is removed. No IRQ routing table is used
>> (irqchip.c and irqcomm.c are not used).
>>
>> Both KVM_CAP_IRQFD & KVM_CAP_IRQFD_RESAMPLE capabilities are exposed
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> This patch serie deprecates the previous serie featuring GSI routing
>> (https://patches.linaro.org/32261/)
>>
>> The patch serie has the following dependencies:
>> - arm/arm64: KVM: Various VGIC cleanups and improvements
>> https://lists.cs.columbia.edu/pipermail/kvmarm/2014-June/009979.html
>> - "KVM: EVENTFD: remove inclusion of irq.h"
>>
>> All pieces can be found on git://git.linaro.org/people/eric.auger/linux.git
>> branch irqfd_norouting_integ_v3
>>
>> This work was tested with Calxeda Midway xgmac main interrupt with
>> qemu-system-arm and QEMU VFIO platform device.
>>
>> v2 -> v3:
>> - removal of irq.h from eventfd.c put in a separate patch to increase
>> visibility
>> - properly expose KVM_CAP_IRQFD capability in arm.c
>> - remove CONFIG_HAVE_KVM_IRQCHIP meaningfull only if irq_comm.c is used
>>
>> v1 -> v2:
>> - rebase on 3.17rc1
>> - move of the dist unlock in process_maintenance
>> - remove of dist lock in __kvm_vgic_sync_hwstate
>> - rewording of the commit message (add resamplefd reference)
>> - remove irq.h
>> ---
>> Documentation/virtual/kvm/api.txt | 5 +++-
>> arch/arm/include/uapi/asm/kvm.h | 3 +++
>> arch/arm/kvm/Kconfig | 4 +--
>> arch/arm/kvm/Makefile | 2 +-
>> arch/arm/kvm/arm.c| 3 +++
>> virt/kvm/arm/vgic.c | 56
>> ---
>> 6 files changed, 65 insertions(+), 8 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt
>> b/Documentation/virtual/kvm/api.txt
>> index beae3fd..8118b12 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2204,7 +2204,7 @@ into the hash PTE second double word).
>> 4.75 KVM_IRQFD
>>
>> Capability: KVM_CAP_IRQFD
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm
>> Type: vm ioctl
>> Parameters: struct kvm_irqfd (in)
>> Returns: 0 on success, -1 on error
>> @@ -2230,6 +2230,9 @@ Note that closing the resamplefd is not sufficient to
>> disable the
>> irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>> and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>
>> +On ARM/arm64 the injected must be a shared peripheral interrupt (SPI).
>> +This means the programmed GIC interrupt ID is gsi+32.
>> +
>
> See above comment.
Hi Christoffer,
sorry which comment do you refer to? wrt your last comment do you
consider PPI injection support is a mandated feature for this patch to
be upstreamable?
>
>> 4.76 KVM_PPC_ALLOCATE_HTAB
>>
>> Capability: KVM_CAP_PPC_ALLOC_HTAB
>> diff --git a/arch/arm/include/uapi/asm/kvm.h
>> b/arch/arm/include/uapi/asm/kvm.h
>> index e6ebdd3..3034c66 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -194,6 +194,9 @@ struct kvm_arch_memory_slot {
>> /* Highest supported SPI, from VGIC_NR_IRQS */
>> #define KVM_ARM_IRQ_GIC_MAX 127
>>
>> +/* One single KVM irqchip, ie. the VGIC */
>> +#define KVM_NR_IRQCHIPS 1
>> +
>> /* PSCI interface */
>> #define KVM_PSCI_FN_BASE0x95c1ba5e
>> #define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
>> diff --git a/arch/arm/kvm/Kc
Re: [RFC v2 4/9] VFIO: platform: handler tests whether the IRQ is forwarded
On 09/11/2014 05:10 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:43PM +0200, Eric Auger wrote:
>> In case the IRQ is forwarded, the VFIO platform IRQ handler does not
>> need to disable the IRQ anymore. In that mode, when the handler completes
>
> add a comma after completes
Hi Christoffer,
ok
>
>> the IRQ is not deactivated but only its priority is lowered.
>>
>> Some other actor (typically a guest) is supposed to deactivate the IRQ,
>> allowing at that time a new physical IRQ to hit.
>>
>> In virtualization use case, the physical IRQ is automatically completed
>> by the interrupt controller when the guest completes the corresponding
>> virtual IRQ.
>>
>> Signed-off-by: Eric Auger
>> ---
>> drivers/vfio/platform/vfio_platform_irq.c | 7 ++-
>> 1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_irq.c
>> b/drivers/vfio/platform/vfio_platform_irq.c
>> index 6768508..1f851b2 100644
>> --- a/drivers/vfio/platform/vfio_platform_irq.c
>> +++ b/drivers/vfio/platform/vfio_platform_irq.c
>> @@ -88,13 +88,18 @@ static irqreturn_t vfio_irq_handler(int irq, void
>> *dev_id)
>> struct vfio_platform_irq *irq_ctx = dev_id;
>> unsigned long flags;
>> int ret = IRQ_NONE;
>> +struct irq_data *d;
>> +bool is_forwarded;
>>
>> spin_lock_irqsave(&irq_ctx->lock, flags);
>>
>> if (!irq_ctx->masked) {
>> ret = IRQ_HANDLED;
>> +d = irq_get_irq_data(irq_ctx->hwirq);
>> +is_forwarded = irqd_irq_forwarded(d);
>>
>> -if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED) {
>> +if (irq_ctx->flags & VFIO_IRQ_INFO_AUTOMASKED &&
>> +!is_forwarded) {
>> disable_irq_nosync(irq_ctx->hwirq);
>> irq_ctx->masked = true;
>> }
>> --
>> 1.9.1
>>
> It makes sense that these needs to be all controlled in the kernel, but
> I'm wondering if it would be cleaner / more correct to clear the
> AUTOMASKED flag when the IRQ is forwarded and have vfio refuse setting
> this flag as long as the irq is forwarded?
If I am not wrong, even if the user sets AUTOMASKED, this info never is
exploited by the vfio platform driver. AUTOMASKED only is set internally
to the driver, on init, for level sensitive IRQs.
It seems to be the same on PCI (for INTx). I do not see anywhere the
user flag curectly copied into a local storage. But I prefer to be
careful ;-)
If confirmed, although the flag value is exposed in the user API, the
user set value never is exploited so this removes the need to check.
the forwarded IRQ modality being fully dynamic currently, then I would
need to update the irq_ctx->flags on each vfio_irq_handler call. I don't
know if its better?
Best Regards
Eric
>
> -Christoffer
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2 5/9] KVM: KVM-VFIO: update user API to program forwarded IRQ
On 09/11/2014 05:10 AM, Christoffer Dall wrote: > On Mon, Sep 01, 2014 at 02:52:44PM +0200, Eric Auger wrote: >> add new device group commands: >> - KVM_DEV_VFIO_DEVICE_FORWARD_IRQ and >> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ >> >> which enable to turn forwarded IRQ mode on/off. >> >> the kvm_arch_forwarded_irq struct embodies a forwarded IRQ >> >> Signed-off-by: Eric Auger >> >> --- >> >> v1 -> v2: >> - struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h >> to include/uapi/linux/kvm.h >> also irq_index renamed into index and guest_irq renamed into gsi >> - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD >> --- >> Documentation/virtual/kvm/devices/vfio.txt | 26 ++ >> include/uapi/linux/kvm.h | 9 + >> 2 files changed, 35 insertions(+) >> >> diff --git a/Documentation/virtual/kvm/devices/vfio.txt >> b/Documentation/virtual/kvm/devices/vfio.txt >> index ef51740..048baa0 100644 >> --- a/Documentation/virtual/kvm/devices/vfio.txt >> +++ b/Documentation/virtual/kvm/devices/vfio.txt >> @@ -13,6 +13,7 @@ VFIO-group is held by KVM. >> >> Groups: >>KVM_DEV_VFIO_GROUP >> + KVM_DEV_VFIO_DEVICE >> >> KVM_DEV_VFIO_GROUP attributes: >>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking >> @@ -20,3 +21,28 @@ KVM_DEV_VFIO_GROUP attributes: >> >> For each, kvm_device_attr.addr points to an int32_t file descriptor >> for the VFIO group. >> + >> +KVM_DEV_VFIO_DEVICE attributes: >> + KVM_DEV_VFIO_DEVICE_FORWARD_IRQ >> + KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ >> + >> +For each, kvm_device_attr.addr points to a kvm_arch_forwarded_irq struct. >> +This user API makes possible to create a special IRQ handling mode, > > KVM_DEV_VFIO_DEVICE_FORWARD_IRQ enables a special IRQ handling mode on > hardware that supports it, OK > >> +where KVM and a VFIO platform driver collaborate to improve IRQ >> +handling performance. >> + >> +'fd represents the file descriptor of a valid VFIO device whose physical > > fd is described out of context here. Can you copy the struct definition > into this document, perhaps right after the "For each, ..." line above. yes sure > >> +IRQ, referenced by its index, is injected into the VM guest irq (gsi). > as a virtual IRQ (specified >by the gsi field) into the >VM. > >> + >> +On FORWARD_IRQ, KVM-VFIO device programs: >When setting the KVM_DEV_VFIO_DEVICE_FORWARD_IRQ attribute, the >KVM-VFIO device tells the host (or VFIO?) to not complete the >physical IRQ, and instead ensures that KVM (or the VM) completes the >physical IRQ. > >> +- the host, to not complete the physical IRQ itself. >> +- the GIC, to automatically complete the physical IRQ when the guest >> + completes the virtual IRQ. > > and drop this bullet form. ok > >> +This avoids trapping the end-of-interrupt for level sensitive IRQ. > > avoid this last line, it's specific to ARM. ok > >> + >> +On UNFORWARD_IRQ, one returns to the mode where the host completes the >When setting the KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ attribute, the >host (VFIO?) will again complete the physical IRQ and KVM will not... > >> +physical IRQ and the guest completes the virtual IRQ. >> + >> +It is up to the caller of this API to make sure the IRQ is not >> +outstanding when the FORWARD/UNFORWARD is called. This could lead to > > outstanding? can you be specific? active? and I should add *physical* IRQ > > don't refer to FOWARD/UNFORWARD, either refer to these attributes by > their full name or use a clear reference in proper English. ok > >> +some inconsistency on who is going to complete the IRQ. > > This sounds like the whole thing is fragile and if userspace doesn't do > things right, IRQ handling of a piece of hardware is going to be > inconsistent? Is this the case? If so, we need some stronger > semantics. If not, this should be rephrased. Actually the KVM-VFIO device rejects any attempt to change the forwarding mode if the physical IRQ is active. So I hope this is robust and will change the explanation. Thanks Eric > >> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h >> index cf3a2ff..8cd7b0e 100644 >> --- a/include/uapi/linux/kvm.h >> +++ b/include/uapi/linux/kvm.h >> @@ -947,6 +947,12 @@ struct kvm
Re: [RFC v2 6/9] VFIO: Extend external user API
On 09/11/2014 05:10 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:45PM +0200, Eric Auger wrote:
>> New functions are added to be called from ARM KVM-VFIO device.
>
> This commit message seems somewhat random. This patch doesn't deal with
> anything ARM specific, it introduces some generic functions that allows
> users external to vfio itself to retrieve information about a vfio
> platform device.
Yes you're right.
>
>>
>> - vfio_device_get_external_user enables to get a vfio device from
>> its fd
>> - vfio_device_put_external_user puts the vfio device
>> - vfio_external_base_device returns the struct device*,
>> useful to access the platform_device
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v1 -> v2:
>>
>> - vfio_external_get_base_device renamed into vfio_external_base_device
>> - vfio_external_get_type removed
>> ---
>> drivers/vfio/vfio.c | 24
>> include/linux/vfio.h | 3 +++
>> 2 files changed, 27 insertions(+)
>>
>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>> index 8e84471..282814e 100644
>> --- a/drivers/vfio/vfio.c
>> +++ b/drivers/vfio/vfio.c
>> @@ -1401,6 +1401,30 @@ void vfio_group_put_external_user(struct vfio_group
>> *group)
>> }
>> EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
>>
>> +struct vfio_device *vfio_device_get_external_user(struct file *filep)
>> +{
>> +struct vfio_device *vdev = filep->private_data;
>> +
>> +if (filep->f_op != &vfio_device_fops)
>> +return ERR_PTR(-EINVAL);
>> +
>> +vfio_device_get(vdev);
>> +return vdev;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_device_get_external_user);
>> +
>> +void vfio_device_put_external_user(struct vfio_device *vdev)
>> +{
>> +vfio_device_put(vdev);
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_device_put_external_user);
>> +
>> +struct device *vfio_external_base_device(struct vfio_device *vdev)
>> +{
>> +return vdev->dev;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_external_base_device);
>> +
>> int vfio_external_user_iommu_id(struct vfio_group *group)
>> {
>> return iommu_group_id(group->iommu_group);
>> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
>> index ffe04ed..bd4b6cb 100644
>> --- a/include/linux/vfio.h
>> +++ b/include/linux/vfio.h
>> @@ -99,6 +99,9 @@ extern void vfio_group_put_external_user(struct vfio_group
>> *group);
>> extern int vfio_external_user_iommu_id(struct vfio_group *group);
>> extern long vfio_external_check_extension(struct vfio_group *group,
>>unsigned long arg);
>> +extern struct vfio_device *vfio_device_get_external_user(struct file
>> *filep);
>> +extern void vfio_device_put_external_user(struct vfio_device *vdev);
>> +extern struct device *vfio_external_base_device(struct vfio_device *vdev);
>>
>> struct pci_dev;
>> #ifdef CONFIG_EEH
>> --
>> 1.9.1
>>
> Looks good to me,
> -Christoffer
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2 7/9] KVM: KVM-VFIO: add new VFIO external API hooks
On 09/11/2014 05:10 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:46PM +0200, Eric Auger wrote:
>> add functions that implement the gateway to the extended
>
> Capital letter when beginning a new sentence. Also the reference to
> 'the extended VFIO API' feels a bit weird. Can't you make your commit
> message a little more descriptive of this patch, something along the
> lines of:
>
> Provide wrapper functions that allows KVM-VFIO device code to get an
> external handle on a struct vfio_device based on a vfio device file
> descriptor. We provide this through three new functions:
>
> (assuming I got this right).
sure
>
>
>
>> external VFIO API:
>> - kvm_vfio_device_get_external_user
>> - kvm_vfio_device_put_external_user
>> - kvm_vfio_external_base_device
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v1 -> v2:
>> - kvm_vfio_external_get_base_device renamed into
>> kvm_vfio_external_base_device
>> - kvm_vfio_external_get_type removed
>> ---
>> arch/arm/include/asm/kvm_host.h | 5 +
>> virt/kvm/vfio.c | 45
>> +
>> 2 files changed, 50 insertions(+)
>>
>> diff --git a/arch/arm/include/asm/kvm_host.h
>> b/arch/arm/include/asm/kvm_host.h
>> index 6dfb404..1aee6bb 100644
>> --- a/arch/arm/include/asm/kvm_host.h
>> +++ b/arch/arm/include/asm/kvm_host.h
>> @@ -171,6 +171,11 @@ void kvm_set_spte_hva(struct kvm *kvm, unsigned long
>> hva, pte_t pte);
>> unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
>> int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
>>
>> +struct vfio_device;
>> +struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep);
>> +void kvm_vfio_device_put_external_user(struct vfio_device *vdev);
>> +struct device *kvm_vfio_external_base_device(struct vfio_device *vdev);
>> +
>> /* We do not have shadow page tables, hence the empty hooks */
>> static inline int kvm_age_hva(struct kvm *kvm, unsigned long hva)
>> {
>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>> index ba1a93f..76dc7a1 100644
>> --- a/virt/kvm/vfio.c
>> +++ b/virt/kvm/vfio.c
>> @@ -59,6 +59,51 @@ static void kvm_vfio_group_put_external_user(struct
>> vfio_group *vfio_group)
>> symbol_put(vfio_group_put_external_user);
>> }
>>
>> +struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep)
>> +{
>> +struct vfio_device *vdev;
>> +struct vfio_device *(*fn)(struct file *);
>> +
>> +fn = symbol_get(vfio_device_get_external_user);
>> +if (!fn)
>> +return ERR_PTR(-EINVAL);
>> +
>> +vdev = fn(filep);
>> +
>> +symbol_put(vfio_device_get_external_user);
>> +
>> +return vdev;
>> +}
>> +
>> +void kvm_vfio_device_put_external_user(struct vfio_device *vdev)
>> +{
>> +void (*fn)(struct vfio_device *);
>> +
>> +fn = symbol_get(vfio_device_put_external_user);
>> +if (!fn)
>> +return;
>> +
>> +fn(vdev);
>> +
>> +symbol_put(vfio_device_put_external_user);
>> +}
>> +
>> +struct device *kvm_vfio_external_base_device(struct vfio_device *vdev)
>> +{
>> +struct device *(*fn)(struct vfio_device *);
>> +struct device *dev;
>> +
>> +fn = symbol_get(vfio_external_base_device);
>> +if (!fn)
>> +return NULL;
>> +
>> +dev = fn(vdev);
>> +
>> +symbol_put(vfio_external_base_device);
>> +
>> +return dev;
>> +}
>> +
>> static bool kvm_vfio_group_is_coherent(struct vfio_group *vfio_group)
>> {
>> long (*fn)(struct vfio_group *, unsigned long);
>> --
>> 1.9.1
>>
>
> otherwise looks good to me!
> -Christoffer
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2 8/9] KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ forwarding control
On 09/11/2014 05:10 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:47PM +0200, Eric Auger wrote:
>> This patch introduces a new KVM_DEV_VFIO_DEVICE attribute.
>>
>> This is a new control channel which enables KVM to cooperate with
>> viable VFIO devices.
>>
>> The kvm-vfio device now holds a list of devices (kvm_vfio_device)
>> in addition to a list of groups (kvm_vfio_group). The new
>> infrastructure enables to check the validity of the VFIO device
>> file descriptor, get and hold a reference to it.
>>
>> The first concrete implemented command is IRQ forward control:
>> KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
>>
>> It consists in programing the VFIO driver and KVM in a consistent manner
>> so that an optimized IRQ injection/completion is set up. Each
>> kvm_vfio_device holds a list of forwarded IRQ. When putting a
>> kvm_vfio_device, the implementation makes sure the forwarded IRQs
>> are set again in the normal handling state (non forwarded).
>
> 'putting a kvm_vfio_device' sounds to like you're golf'ing :)
>
> When a kvm_vfio_device is released?
sure
>
>>
>> The forwarding programmming is architecture specific, embodied by the
>> kvm_arch_set_fwd_state function. Its implementation is given in a
>> separate patch file.
>
> I would drop the last sentence and instead indicate that this is handled
> properly when the architecture does not support such a feature.
ok
>
>>
>> The forwarding control modality is enabled by the
>> __KVM_HAVE_ARCH_KVM_VFIO_FORWARD define.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v1 -> v2:
>> - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> - original patch file separated into 2 parts: generic part moved in vfio.c
>> and ARM specific part(kvm_arch_set_fwd_state)
>> ---
>> include/linux/kvm_host.h | 27 +++
>> virt/kvm/vfio.c | 452
>> ++-
>> 2 files changed, 477 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index a4c33b3..24350dc 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1065,6 +1065,21 @@ struct kvm_device_ops {
>>unsigned long arg);
>> };
>>
>> +enum kvm_fwd_irq_action {
>> +KVM_VFIO_IRQ_SET_FORWARD,
>> +KVM_VFIO_IRQ_SET_NORMAL,
>> +KVM_VFIO_IRQ_CLEANUP,
>
> This is KVM internal API, so it would probably be good to document this.
> Especially the CLEANUP bit worries me, see below.
I will document it
>
>> +};
>> +
>> +/* internal structure describing a forwarded IRQ */
>> +struct kvm_fwd_irq {
>> +struct list_head link;
>
> this list entry is local to the kvm vfio device, right? that means you
> probably want a struct with just the below fields, and then have a
> containing struct in the generic device file, private to it's logic.
I will introduce 2 separate structs
>
>> +__u32 index; /* platform device irq index */
>> +__u32 hwirq; /*physical IRQ */
>> +__u32 gsi; /* virtual IRQ */
>> +struct kvm_vcpu *vcpu; /* vcpu to inject into*/
>> +};
>> +
>> void kvm_device_get(struct kvm_device *dev);
>> void kvm_device_put(struct kvm_device *dev);
>> struct kvm_device *kvm_device_from_filp(struct file *filp);
>> @@ -1075,6 +1090,18 @@ extern struct kvm_device_ops kvm_vfio_ops;
>> extern struct kvm_device_ops kvm_arm_vgic_v2_ops;
>> extern struct kvm_device_ops kvm_flic_ops;
>>
>> +#ifdef __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> +int kvm_arch_set_fwd_state(struct kvm_fwd_irq *pfwd,
>
> what's the 'p' in pfwd?
will rename
>
>> + enum kvm_fwd_irq_action action);
>> +
>> +#else
>> +static inline int kvm_arch_set_fwd_state(struct kvm_fwd_irq *pfwd,
>> + enum kvm_fwd_irq_action action)
>> +{
>> +return 0;
>> +}
>> +#endif
>> +
>> #ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
>>
>> static inline void kvm_vcpu_set_in_spin_loop(struct kvm_vcpu *vcpu, bool
>> val)
>> diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
>> index 76dc7a1..e4a81c4 100644
>> --- a/virt/kvm/vfio.c
>> +++ b/virt/kvm/vfio.c
>> @@ -18,14 +18,24 @@
>> #include
>> #include
>> #include
>> +#include
>>
>> struct kvm_vfio_group {
>> struct list_head node;
>> struc
Re: [RFC v2 8/9] KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ forwarding control
On 09/11/2014 07:05 AM, Alex Williamson wrote:
> On Thu, 2014-09-11 at 05:10 +0200, Christoffer Dall wrote:
>> On Mon, Sep 01, 2014 at 02:52:47PM +0200, Eric Auger wrote:
>>> This patch introduces a new KVM_DEV_VFIO_DEVICE attribute.
>>>
>>> This is a new control channel which enables KVM to cooperate with
>>> viable VFIO devices.
>>>
>>> The kvm-vfio device now holds a list of devices (kvm_vfio_device)
>>> in addition to a list of groups (kvm_vfio_group). The new
>>> infrastructure enables to check the validity of the VFIO device
>>> file descriptor, get and hold a reference to it.
>>>
>>> The first concrete implemented command is IRQ forward control:
>>> KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
>>>
>>> It consists in programing the VFIO driver and KVM in a consistent manner
>>> so that an optimized IRQ injection/completion is set up. Each
>>> kvm_vfio_device holds a list of forwarded IRQ. When putting a
>>> kvm_vfio_device, the implementation makes sure the forwarded IRQs
>>> are set again in the normal handling state (non forwarded).
>>
>> 'putting a kvm_vfio_device' sounds to like you're golf'ing :)
>>
>> When a kvm_vfio_device is released?
>>
>>>
>>> The forwarding programmming is architecture specific, embodied by the
>>> kvm_arch_set_fwd_state function. Its implementation is given in a
>>> separate patch file.
>>
>> I would drop the last sentence and instead indicate that this is handled
>> properly when the architecture does not support such a feature.
>>
>>>
>>> The forwarding control modality is enabled by the
>>> __KVM_HAVE_ARCH_KVM_VFIO_FORWARD define.
>>>
>>> Signed-off-by: Eric Auger
>>>
>>> ---
>>>
>>> v1 -> v2:
>>> - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>>> - original patch file separated into 2 parts: generic part moved in vfio.c
>>> and ARM specific part(kvm_arch_set_fwd_state)
>>> ---
>>> include/linux/kvm_host.h | 27 +++
>>> virt/kvm/vfio.c | 452
>>> ++-
>>> 2 files changed, 477 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>>> index a4c33b3..24350dc 100644
>>> --- a/include/linux/kvm_host.h
>>> +++ b/include/linux/kvm_host.h
>>> @@ -1065,6 +1065,21 @@ struct kvm_device_ops {
>>> unsigned long arg);
>>> };
>>>
>>> +enum kvm_fwd_irq_action {
>>> + KVM_VFIO_IRQ_SET_FORWARD,
>>> + KVM_VFIO_IRQ_SET_NORMAL,
>>> + KVM_VFIO_IRQ_CLEANUP,
>>
>> This is KVM internal API, so it would probably be good to document this.
>> Especially the CLEANUP bit worries me, see below.
>
> This also doesn't match the user API, which is simply FORWARD/UNFORWARD.
Hi Alex,
will change that.
> Extra states worry me too.
I tried to explained the 2 motivations behind. Please let me know if it
makes sense.
>
>>> +};
>>> +
>>> +/* internal structure describing a forwarded IRQ */
>>> +struct kvm_fwd_irq {
>>> + struct list_head link;
>>
>> this list entry is local to the kvm vfio device, right? that means you
>> probably want a struct with just the below fields, and then have a
>> containing struct in the generic device file, private to it's logic.
>
> Yes, this is part of the abstraction problem.
OK will fix that.
>
>>> + __u32 index; /* platform device irq index */
>
> This is a vfio_device irq_index, but vfio_devices support indexes and
> sub-indexes. At this level the API should match vfio, not the specifics
> of platform devices not supporting sub-index.
I will add sub-indexes then.
>
>>> + __u32 hwirq; /*physical IRQ */
>>> + __u32 gsi; /* virtual IRQ */
>>> + struct kvm_vcpu *vcpu; /* vcpu to inject into*/
>
> Not sure I understand why vcpu is necessary.
vcpu is used when providing the physical IRQ/virtual IRQ mapping to the
virtual GIC. I can remove it from and add a vcpu struct * param to
kvm_arch_set_fwd_state if you prefer.
Also I see a 'get' in the code below, but not a 'put'.
Sorry I do not understand your comment here? What 'get' do you mention?
>
>>> +};
>>> +
>>> void kvm_device_get(struct kvm_device *dev);
>>> void kvm_device_put(struct kvm_device *dev);
>>> struct kvm_device
Re: [RFC v2 2/9] KVM: ARM: VGIC: add forwarded irq rbtree lock
On 09/11/2014 05:09 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:41PM +0200, Eric Auger wrote:
>> add a lock related to the rb tree manipulation. The rb tree can be
>
> Ok, I can't hold myself back any longer.
Please begin sentences with a
> capital letter. You don't do this in French? :)
Hi Christoffer,
yep that's understood ;-) Definitively we do. Just that I am discovering
it is common too in commits and comments ;-)
>
>> searched in one thread (irqfd handler for instance) and map/unmap
>> happen in another.
>>
>> Signed-off-by: Eric Auger
>> ---
>> include/kvm/arm_vgic.h | 1 +
>> virt/kvm/arm/vgic.c| 46 +-
>> 2 files changed, 38 insertions(+), 9 deletions(-)
>>
>> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
>> index 743020f..3da244f 100644
>> --- a/include/kvm/arm_vgic.h
>> +++ b/include/kvm/arm_vgic.h
>> @@ -177,6 +177,7 @@ struct vgic_dist {
>> unsigned long irq_pending_on_cpu;
>>
>> struct rb_root irq_phys_map;
>> +spinlock_t rb_tree_lock;
>> #endif
>> };
>>
>> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
>> index 8ef495b..dbc2a5a 100644
>> --- a/virt/kvm/arm/vgic.c
>> +++ b/virt/kvm/arm/vgic.c
>> @@ -1630,9 +1630,15 @@ static struct rb_root *vgic_get_irq_phys_map(struct
>> kvm_vcpu *vcpu,
>>
>> int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
>> {
>> -struct rb_root *root = vgic_get_irq_phys_map(vcpu, virt_irq);
>> -struct rb_node **new = &root->rb_node, *parent = NULL;
>> +struct rb_root *root;
>> +struct rb_node **new, *parent = NULL;
>> struct irq_phys_map *new_map;
>> +struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +
>> +spin_lock(&dist->rb_tree_lock);
>> +
>> +root = vgic_get_irq_phys_map(vcpu, virt_irq);
>> +new = &root->rb_node;
>>
>> /* Boilerplate rb_tree code */
>> while (*new) {
>> @@ -1644,13 +1650,17 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
>> virt_irq, int phys_irq)
>> new = &(*new)->rb_left;
>> else if (this->virt_irq > virt_irq)
>> new = &(*new)->rb_right;
>> -else
>> +else {
>> +spin_unlock(&dist->rb_tree_lock);
>> return -EEXIST;
>> +}
>
> can you initialize a ret variable to -EEXIST in the beginning of this
> function, and add an out label above the unlock below, replace this
> multi-line statement with a goto out, and set ret = 0 after the while
> loop?
sure
>
>> }
>>
>> new_map = kzalloc(sizeof(*new_map), GFP_KERNEL);
>> -if (!new_map)
>> +if (!new_map) {
>> +spin_unlock(&dist->rb_tree_lock);
>> return -ENOMEM;
>
> then this becomes ret = -ENOMEM; goto out;
OK
>
>> +}
>>
>> new_map->virt_irq = virt_irq;
>> new_map->phys_irq = phys_irq;
>> @@ -1658,6 +1668,8 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
>> virt_irq, int phys_irq)
>> rb_link_node(&new_map->node, parent, new);
>> rb_insert_color(&new_map->node, root);
>>
>> +spin_unlock(&dist->rb_tree_lock);
>> +
>
> aren't you allocating memory with GFP_KERNEL while holding a spinlock
> here?
oups. Thanks for noticing. I Will move the lock.
>
>> return 0;
>> }
>>
>> @@ -1685,24 +1697,39 @@ static struct irq_phys_map
>> *vgic_irq_map_search(struct kvm_vcpu *vcpu,
>>
>> int vgic_get_phys_irq(struct kvm_vcpu *vcpu, int virt_irq)
>> {
>> -struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
>> +struct irq_phys_map *map;
>> +struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +int ret;
>> +
>> +spin_lock(&dist->rb_tree_lock);
>> +map = vgic_irq_map_search(vcpu, virt_irq);
>>
>> if (map)
>> -return map->phys_irq;
>> +ret = map->phys_irq;
>> +else
>> +ret = -ENOENT;
>
> initialize ret to -ENOENT and avoid the else statement.
ok
>
>> +
>> +spin_unlock(&dist->rb_tree_lock);
>> +return ret;
>>
>> -return -ENOENT;
>> }
>>
>> int vgic_unmap_phys_irq(stru
Re: [RFC v2 1/9] KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded IRQ
On 09/11/2014 05:09 AM, Christoffer Dall wrote:
> On Mon, Sep 01, 2014 at 02:52:40PM +0200, Eric Auger wrote:
>> Fix multiple injection of level sensitive forwarded IRQs.
>> With current code, the second injection fails since the state bitmaps
>> are not reset (process_maintenance is not called anymore).
>> New implementation consists in fully bypassing the vgic state
>> management for forwarded IRQ (checks are ignored in
>> vgic_update_irq_pending). This obviously assumes the forwarded IRQ is
>> injected from kernel side.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> It was attempted to reset the states in __kvm_vgic_sync_hwstate, checking
>> the emptied LR of forwarded IRQ. However surprisingly this solution does
>> not seem to work. Some times, a new forwarded IRQ injection is observed
>> while the LR of the previous instance was not observed as empty.
>
> hmmm, concerning. It would probably have been helpful overall if you
> could start by describing the problem with the current implementation in
> the commit message, and then explain the fix...
>
>>
>> v1 -> v2:
>> - fix vgic state bypass in vgic_queue_hwirq
>> ---
>> virt/kvm/arm/vgic.c | 13 ++---
>> 1 file changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
>> index 0007300..8ef495b 100644
>> --- a/virt/kvm/arm/vgic.c
>> +++ b/virt/kvm/arm/vgic.c
>> @@ -1259,7 +1259,9 @@ static bool vgic_queue_sgi(struct kvm_vcpu *vcpu, int
>> irq)
>>
>> static bool vgic_queue_hwirq(struct kvm_vcpu *vcpu, int irq)
>> {
>> -if (vgic_irq_is_queued(vcpu, irq))
>> +bool is_forwarded = (vgic_get_phys_irq(vcpu, irq) > 0);
>
> can you create a static function to factor this vgic_get_phys_irq check out,
> please?
yes sure
>
>> +
>> +if (vgic_irq_is_queued(vcpu, irq) && !is_forwarded)
>> return true; /* level interrupt, already queued */
>
> so essentially if an IRQ is already on a LR so we shouldn't resample the
> line, then we still resample the line if the IRQ is forwarded?
>
> I think you need to explain this, both to me here, and also in the code
> by moving the comment following the return statement above the check and
> comment this clearly.
Well, I admit it may look a bit pushy! When we discussed this issue with
Marc, the outcome was that the vgic states were not accurate with
forwarded IRQs and VGIC state may be fully bypassed. Since the first
injection still sets the state - and I did not want to modify this - the
2d one would fail due to that check, and the validate_injection. May be
cleaner to not update the states when injecting the fwd irq too.
>
>>
>> if (vgic_queue_irq(vcpu, 0, irq)) {
>> @@ -1517,14 +1519,18 @@ static bool vgic_update_irq_pending(struct kvm *kvm,
>> int cpuid,
>> int edge_triggered, level_triggered;
>> int enabled;
>> bool ret = true;
>> +bool is_forwarded;
>>
>> spin_lock(&dist->lock);
>>
>> vcpu = kvm_get_vcpu(kvm, cpuid);
>> +is_forwarded = (vgic_get_phys_irq(vcpu, irq_num) > 0);
>
> use your new function here as well.
ok
>
>> +
>> edge_triggered = vgic_irq_is_edge(vcpu, irq_num);
>> level_triggered = !edge_triggered;
>>
>> -if (!vgic_validate_injection(vcpu, irq_num, level)) {
>> +if (!is_forwarded &&
>> +!vgic_validate_injection(vcpu, irq_num, level)) {
>
> I don't see the rationale here either. If an IRQ is forwarded, why do
> you need to do anything if the condition of the line hasn't changed for
> a level-triggered IRQ or if you have a falling edge on an edge-triggered
> IRQ (assuming active-HIGH)?
To me this even cannot cannot happen. a second fwd irq can only hit if
the same virtual IRQ was completed and completed the corresponding phys
IRQ. Still the problem is that on the 1st injection we updated the VGIC
state. I aknowledge this is a hack to work around the 1st injection
update the state and nothing reset them. So on subsequent injections, -
and even on the 1st one- I never check the state.
>
>> ret = false;
>> goto out;
>> }
>> @@ -1557,7 +1563,8 @@ static bool vgic_update_irq_pending(struct kvm *kvm,
>> int cpuid,
>> goto out;
>> }
>>
>> -if (level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
>> +if (!is_forwarded &&
>> +level_triggered && vgic_irq_is_queued(vcpu, irq_num)) {
>
> So here it's making sense
Re: [RFC PATCH 1/4] ARM: KVM: on unhandled IO mem abort, route the call to the KVM MMIO bus
On 11/13/2014 04:31 PM, Andre Przywara wrote:
>
>
> On 13/11/14 15:02, Nikolay Nikolaev wrote:
>> On Thu, Nov 13, 2014 at 4:23 PM, Eric Auger wrote:
>>> On 11/13/2014 03:16 PM, Eric Auger wrote:
>>>> On 11/13/2014 11:45 AM, Nikolay Nikolaev wrote:
>>>>> On Mon, Nov 10, 2014 at 6:27 PM, Christoffer Dall
>>>>> wrote:
>>>>>> On Mon, Nov 10, 2014 at 05:09:07PM +0200, Nikolay Nikolaev wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> On Fri, Mar 28, 2014 at 9:09 PM, Christoffer Dall
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Thu, Mar 13, 2014 at 04:57:26PM +0100, Antonios Motakis wrote:
>>>>>>>>> On an unhandled IO memory abort, use the kvm_io_bus_* API in order to
>>>>>>>>> handle the MMIO access through any registered read/write callbacks.
>>>>>>>>> This
>>>>>>>>> is a dependency for eventfd support (ioeventfd and irqfd).
>>>>>>>>>
>>>>>>>>> However, accesses to the VGIC are still left implemented
>>>>>>>>> independently,
>>>>>>>>> since the kvm_io_bus_* API doesn't pass the VCPU pointer doing the
>>>>>>>>> access.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Antonios Motakis
>>>>>>>>> Signed-off-by: Nikolay Nikolaev
>>>>>>>>> ---
>>>>>>>>> arch/arm/kvm/mmio.c | 32
>>>>>>>>> virt/kvm/arm/vgic.c | 5 -
>>>>>>>>> 2 files changed, 36 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/arch/arm/kvm/mmio.c b/arch/arm/kvm/mmio.c
>>>>>>>>> index 4cb5a93..1d17831 100644
>>>>>>>>> --- a/arch/arm/kvm/mmio.c
>>>>>>>>> +++ b/arch/arm/kvm/mmio.c
>>>>>>>>> @@ -162,6 +162,35 @@ static int decode_hsr(struct kvm_vcpu *vcpu,
>>>>>>>>> phys_addr_t fault_ipa,
>>>>>>>>> return 0;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> +/**
>>>>>>>>> + * kvm_handle_mmio - handle an in-kernel MMIO access
>>>>>>>>> + * @vcpu:pointer to the vcpu performing the access
>>>>>>>>> + * @run: pointer to the kvm_run structure
>>>>>>>>> + * @mmio:pointer to the data describing the access
>>>>>>>>> + *
>>>>>>>>> + * returns true if the MMIO access has been performed in kernel
>>>>>>>>> space,
>>>>>>>>> + * and false if it needs to be emulated in user space.
>>>>>>>>> + */
>>>>>>>>> +static bool handle_kernel_mmio(struct kvm_vcpu *vcpu, struct kvm_run
>>>>>>>>> *run,
>>>>>>>>> + struct kvm_exit_mmio *mmio)
>>>>>>>>> +{
>>>>>>>>> + int ret;
>>>>>>>>> + if (mmio->is_write) {
>>>>>>>>> + ret = kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS,
>>>>>>>>> mmio->phys_addr,
>>>>>>>>> + mmio->len, &mmio->data);
>>>>>>>>> +
>>>>>>>>> + } else {
>>>>>>>>> + ret = kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS,
>>>>>>>>> mmio->phys_addr,
>>>>>>>>> + mmio->len, &mmio->data);
>>>>>>>>> + }
>>>>>>>>> + if (!ret) {
>>>>>>>>> + kvm_prepare_mmio(run, mmio);
>>>>>>>>> + kvm_handle_mmio_return(vcpu, run);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + return !ret;
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run,
>>>>>>>>>phys_addr_t fault_ipa)
>>>>>>>>>
Re: [RFC v2 0/9] KVM-VFIO IRQ forward control
Hi Feng, I will submit a PATCH v3 release end of this week. Best Regards Eric On 11/17/2014 12:25 PM, Wu, Feng wrote: > > >> -Original Message- >> From: linux-kernel-ow...@vger.kernel.org >> [mailto:linux-kernel-ow...@vger.kernel.org] On Behalf Of Alex Williamson >> Sent: Thursday, September 11, 2014 1:10 PM >> To: Christoffer Dall >> Cc: Eric Auger; eric.au...@st.com; marc.zyng...@arm.com; >> linux-arm-ker...@lists.infradead.org; kvm...@lists.cs.columbia.edu; >> kvm@vger.kernel.org; joel.sch...@amd.com; kim.phill...@freescale.com; >> pau...@samba.org; g...@kernel.org; pbonz...@redhat.com; >> linux-ker...@vger.kernel.org; patc...@linaro.org; will.dea...@arm.com; >> a.mota...@virtualopensystems.com; a.r...@virtualopensystems.com; >> john.li...@huawei.com >> Subject: Re: [RFC v2 0/9] KVM-VFIO IRQ forward control >> >> On Thu, 2014-09-11 at 05:10 +0200, Christoffer Dall wrote: >>> On Tue, Sep 02, 2014 at 03:05:41PM -0600, Alex Williamson wrote: >>>> On Mon, 2014-09-01 at 14:52 +0200, Eric Auger wrote: >>>>> This RFC proposes an integration of "ARM: Forwarding physical >>>>> interrupts to a guest VM" (http://lwn.net/Articles/603514/) in >>>>> KVM. >>>>> >>>>> It enables to transform a VFIO platform driver IRQ into a forwarded >>>>> IRQ. The direct benefit is that, for a level sensitive IRQ, a VM >>>>> switch can be avoided on guest virtual IRQ completion. Before this >>>>> patch, a maintenance IRQ was triggered on the virtual IRQ completion. >>>>> >>>>> When the IRQ is forwarded, the VFIO platform driver does not need to >>>>> disable the IRQ anymore. Indeed when returning from the IRQ handler >>>>> the IRQ is not deactivated. Only its priority is lowered. This means >>>>> the same IRQ cannot hit before the guest completes the virtual IRQ >>>>> and the GIC automatically deactivates the corresponding physical IRQ. >>>>> >>>>> Besides, the injection still is based on irqfd triggering. The only >>>>> impact on irqfd process is resamplefd is not called anymore on >>>>> virtual IRQ completion since this latter becomes "transparent". >>>>> >>>>> The current integration is based on an extension of the KVM-VFIO >>>>> device, previously used by KVM to interact with VFIO groups. The >>>>> patch serie now enables KVM to directly interact with a VFIO >>>>> platform device. The VFIO external API was extended for that purpose. >>>>> >>>>> Th KVM-VFIO device can get/put the vfio platform device, check its >>>>> integrity and type, get the IRQ number associated to an IRQ index. >>>>> >>>>> The IRQ forward programming is architecture specific (virtual interrupt >>>>> controller programming basically). However the whole infrastructure is >>>>> kept generic. >>>>> >>>>> from a user point of view, the functionality is provided through new >>>>> KVM-VFIO device commands, >> KVM_DEV_VFIO_DEVICE_(UN)FORWARD_IRQ >>>>> and the capability can be checked with KVM_HAS_DEVICE_ATTR. >>>>> Assignment can only be changed when the physical IRQ is not active. >>>>> It is the responsability of the user to do this check. >>>>> >>>>> This patch serie has the following dependencies: >>>>> - "ARM: Forwarding physical interrupts to a guest VM" >>>>> (http://lwn.net/Articles/603514/) in >>>>> - [PATCH v3] irqfd for ARM >>>>> - and obviously the VFIO platform driver serie: >>>>> [RFC PATCH v6 00/20] VFIO support for platform devices on ARM >>>>> https://www.mail-archive.com/kvm@vger.kernel.org/msg103247.html >>>>> >>>>> Integrated pieces can be found at >>>>> ssh://git.linaro.org/people/eric.auger/linux.git >>>>> on branch 3.17rc3_irqfd_forward_integ_v2 >>>>> >>>>> This was was tested on Calxeda Midway, assigning the xgmac main IRQ. >>>>> >>>>> v1 -> v2: >>>>> - forward control is moved from architecture specific file into generic >>>>> vfio.c module. >>>>> only kvm_arch_set_fwd_state remains architecture specific >>>>> - integrate Kim's patch which enables KVM-VFIO for ARM >>>>> - fix vgic state bypass in vgic_queue_h
Re: [PATCH v9 06/19] vfio/platform: return info for bound device
On 11/20/2014 03:10 PM, Antonios Motakis wrote:
> On Wed, Nov 12, 2014 at 5:36 PM, Alex Williamson
> wrote:
>> On Wed, 2014-11-12 at 11:32 +0100, Eric Auger wrote:
>>> On 10/27/2014 07:07 PM, Antonios Motakis wrote:
>>>> A VFIO userspace driver will start by opening the VFIO device
>>>> that corresponds to an IOMMU group, and will use the ioctl interface
>>>> to get the basic device info, such as number of memory regions and
>>>> interrupts, and their properties. This patch enables the
>>>> VFIO_DEVICE_GET_INFO ioctl call.
>>>>
>>>> Signed-off-by: Antonios Motakis
>>>> ---
>>>> drivers/vfio/platform/vfio_platform_common.c | 23 ---
>>>> 1 file changed, 20 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/drivers/vfio/platform/vfio_platform_common.c
>>>> b/drivers/vfio/platform/vfio_platform_common.c
>>>> index e0fdbc8..cb20526 100644
>>>> --- a/drivers/vfio/platform/vfio_platform_common.c
>>>> +++ b/drivers/vfio/platform/vfio_platform_common.c
>>>> @@ -43,10 +43,27 @@ static int vfio_platform_open(void *device_data)
>>>> static long vfio_platform_ioctl(void *device_data,
>>>>unsigned int cmd, unsigned long arg)
>>>> {
>>>> - if (cmd == VFIO_DEVICE_GET_INFO)
>>>> - return -EINVAL;
>>>> + struct vfio_platform_device *vdev = device_data;
>>>> + unsigned long minsz;
>>>> +
>>>> + if (cmd == VFIO_DEVICE_GET_INFO) {
>>>> + struct vfio_device_info info;
>>>> +
>>>> + minsz = offsetofend(struct vfio_device_info, num_irqs);
>>>> +
>>>> + if (copy_from_user(&info, (void __user *)arg, minsz))
>>>> + return -EFAULT;
>>>> +
>>>> + if (info.argsz < minsz)
>>>> + return -EINVAL;
>>>> +
>>>> + info.flags = vdev->flags;
>>>> + info.num_regions = 0;
>>>> + info.num_irqs = 0;
>>> Seems a bit weird to me to enable the modality but returning zeroed
>>> values. Shouldn't we put that patch after VFIO_DEVICE_GET_REGION_INFO
>>> and VFIO_DEVICE_GET_IRQ_INFO ones?
>>
>> I actually like how Antonios has started from a base framework, exposing
>> a device but none of the resources and then incrementally adds each
>> component. It's also a good showcase of the VFIO ABI that we can do
>> things like this. Thanks,
>
> I also agree with Alex with this. But of course I'm not married with
> any particular splitting style, in case we decide to change this.
Hi Antonios,
please keep as is. I also learn each day about splitting style ;-)
Best Regards
Eric
>
>>
>> Alex
>>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 1/2] vfio: Add new interrupt group for VFIO
On 11/20/2014 04:53 PM, Alex Williamson wrote:
> On Thu, 2014-11-20 at 17:05 +0800, Feng Wu wrote:
>> Add new group KVM_DEV_VFIO_INTERRUPT and command
>> KVM_DEV_VFIO_DEVIE_POSTING_IRQ related to it.
>>
>> This is used for VT-d Posted-Interrupts setup.
>
> Eric proposed an interface for ARM forwarded interrupts[1] using group
> KVM_DEV_VFIO_DEVICE with attributes KVM_DEV_VFIO_DEVICE_ASSIGN_IRQ and
> KVM_DEV_VFIO_DEVICE_DEASSIGN_IRQ. Why are we proposing yet another
> group and attributes here? Why can't we re-use the ones Eric proposes?
Hi Alex, Feng
I share your point of view about the KVM_DEV_VFIO_DEVICE group. For the
attribute (renamed KVM_DEV_VFIO_DEVICE_FORWARD_IRQ in RFC v2) the issue
is I specify the kvm_device_attr.addr points to a kvm_arch_forwarded_irq
struct. Feng needs another struct - kvm_posted_intr -. An alternative is
to merge both structs if it makes sense.
Best Regards
Eric
>
> [1] https://lkml.org/lkml/2014/8/25/258
>
>> Signed-off-by: Feng Wu
>> ---
>> Documentation/virtual/kvm/devices/vfio.txt |8
>> include/uapi/linux/kvm.h | 14 ++
>> 2 files changed, 22 insertions(+), 0 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
>> b/Documentation/virtual/kvm/devices/vfio.txt
>> index ef51740..bd99176 100644
>> --- a/Documentation/virtual/kvm/devices/vfio.txt
>> +++ b/Documentation/virtual/kvm/devices/vfio.txt
>> @@ -13,6 +13,7 @@ VFIO-group is held by KVM.
>>
>> Groups:
>>KVM_DEV_VFIO_GROUP
>> + KVM_DEV_VFIO_INTERRUPT
>>
>> KVM_DEV_VFIO_GROUP attributes:
>>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
>> @@ -20,3 +21,10 @@ KVM_DEV_VFIO_GROUP attributes:
>>
>> For each, kvm_device_attr.addr points to an int32_t file descriptor
>> for the VFIO group.
>> +
>> +KVM_DEV_VFIO_INTERRUPT attributes:
>> + KVM_DEV_VFIO_INTERRUPT_POSTING_IRQ: Set up the interrupt configuration for
>> +VT-d Posted-Interrrupts
>> +
>> +For each, kvm_device_attr.addr points to struct kvm_posted_intr, which
>> +include the needed information for VT-d Posted-Interrupts setup.
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 6076882..5544fcc 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -943,9 +943,23 @@ struct kvm_device_attr {
>> __u64 addr; /* userspace address of attr data */
>> };
>>
>> +struct virq_info {
>> +__u32 index; /* index of the msi/msix entry */
>> +int virq; /* virq of the interrupt */
>> +};
>> +
>> +struct kvm_posted_intr {
>> +__u32 fd; /* file descriptor of the VFIO device */
>> +__u32 count;
>> +boolmsix;
>
> Note that MSI-X (as opposed to MSI) is a PCI concept. Being a VFIO
> interface this should operate at VFIO IRQ index and sub-index.
>
>> +struct virq_info virq_info[0];
>> +};
>> +
>> #define KVM_DEV_VFIO_GROUP 1
>> #define KVM_DEV_VFIO_GROUP_ADD1
>> #define KVM_DEV_VFIO_GROUP_DEL2
>> +#define KVM_DEV_VFIO_INTERRUPT 2
>> +#define KVM_DEV_VFIO_INTERRUPT_POSTING_IRQ1
>>
>> enum kvm_device_type {
>> KVM_DEV_TYPE_FSL_MPIC_20= 1,
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v1 2/2] vfio: Add VFIO API vfio_msi_get_irq
On 11/20/2014 05:05 PM, Alex Williamson wrote:
> On Thu, 2014-11-20 at 17:05 +0800, Feng Wu wrote:
>> This API returns the host irq for the MSI/MSI-X interrrupts.
>>
>> Signed-off-by: Feng Wu
>> ---
>> drivers/vfio/pci/vfio_pci.c | 10 ++
>> include/linux/vfio.h|2 ++
>> 2 files changed, 12 insertions(+), 0 deletions(-)
>>
>> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
>> index 9558da3..4fb9828 100644
>> --- a/drivers/vfio/pci/vfio_pci.c
>> +++ b/drivers/vfio/pci/vfio_pci.c
>> @@ -1009,6 +1009,16 @@ put_devs:
>> kfree(devs.devices);
>> }
>>
>> +unsigned int vfio_msi_get_irq(struct vfio_device *device, int vector, bool
>> msix)
>> +{
>> +struct vfio_pci_device *vdev =
>> +(struct vfio_pci_device *)vfio_device_data(device);
>> +struct pci_dev *pdev = vdev->pdev;
>> +
>> +return msix ? vdev->msix[vector].vector : pdev->irq + vector;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_msi_get_irq);
>> +
>
> It's difficult to see what you're going to do with this, but it's making
> the assumption that the vfio_device is a vfio_pci_device, which is the
> wrong starting point. Eric proposed[1] some VFIO external user
> interface extensions, that have since been refined in discussion that
> allows the external user (KVM in this case) access to the struct device
> where it can get the IRQ vectors itself. We either need to follow that
> same path for posted interrupts, teaching KVM about struct pci_device in
> order to extract the IRQ info, or we need to create another vfio
> external user interface extension that can return IRQ info regardless of
> the device type. Thanks,
>
> Alex
>
>
> [1] https://lkml.org/lkml/2014/8/25/257
Here is the v2: https://lkml.org/lkml/2014/9/1/347
Best Regards
Eric
> http://lwn.net/Articles/610087/
>> static void __exit vfio_pci_cleanup(void)
>> {
>> pci_unregister_driver(&vfio_pci_driver);
>> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
>> index d320411..007ca55 100644
>> --- a/include/linux/vfio.h
>> +++ b/include/linux/vfio.h
>> @@ -92,6 +92,8 @@ extern void vfio_unregister_iommu_driver(
>> /*
>> * External user API
>> */
>> +extern unsigned int vfio_msi_get_irq(struct vfio_device *device, int vector,
>> + bool msix);
>> extern struct vfio_group *vfio_group_get_external_user(struct file *filep);
>> extern void vfio_group_put_external_user(struct vfio_group *group);
>> extern int vfio_external_user_iommu_id(struct vfio_group *group);
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v1] ARM/ARM64: support KVM_IOEVENTFD
Hi Ming,
for your information there is a series written by Antonios (added in CC)
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-March/008416.html
exactly on the same topic.
The thread was reactivated by Nikolay latterly on Nov (see
http://www.gossamer-threads.com/lists/linux/kernel/1886716?page=last).
I am also convinced we must progress on ioeventfd topic concurrently
with irqfd one. What starting point do we use then for further comments?
Best Regards
Eric
On 11/19/2014 06:16 AM, Ming Lei wrote:
> From Documentation/virtual/kvm/api.txt, all ARCHs should support
> ioeventfd.
>
> Also ARM VM has supported PCI bus already, and ARM64 will do too,
> ioeventfd is required for some popular devices, like virtio-blk
> and virtio-scsi dataplane in QEMU.
>
> Without this patch, virtio-blk-pci dataplane can't work in QEMU.
>
> This patch has been tested on both ARM and ARM64.
>
> Signed-off-by: Ming Lei
> ---
> v1:
> - make eventfd.o built in ARM64
> arch/arm/kvm/Kconfig|1 +
> arch/arm/kvm/Makefile |2 +-
> arch/arm/kvm/arm.c |1 +
> arch/arm/kvm/mmio.c | 19 +++
> arch/arm64/kvm/Kconfig |1 +
> arch/arm64/kvm/Makefile |2 +-
> 6 files changed, 24 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 466bd29..25bd83a 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -23,6 +23,7 @@ config KVM
> select HAVE_KVM_CPU_RELAX_INTERCEPT
> select KVM_MMIO
> select KVM_ARM_HOST
> + select HAVE_KVM_EVENTFD
> depends on ARM_VIRT_EXT && ARM_LPAE
> ---help---
> Support hosting virtualized guest machines. You will also
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index f7057ed..859db09 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
> AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
>
> KVM := ../../../virt/kvm
> -kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
> +kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o
>
> obj-y += kvm-arm.o init.o interrupts.o
> obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 9e193c8..d90d989 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -172,6 +172,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long
> ext)
> case KVM_CAP_IRQCHIP:
> r = vgic_present;
> break;
> + case KVM_CAP_IOEVENTFD:
> case KVM_CAP_DEVICE_CTRL:
> case KVM_CAP_USER_MEMORY:
> case KVM_CAP_SYNC_MMU:
> diff --git a/arch/arm/kvm/mmio.c b/arch/arm/kvm/mmio.c
> index 4cb5a93..ee332a7 100644
> --- a/arch/arm/kvm/mmio.c
> +++ b/arch/arm/kvm/mmio.c
> @@ -162,6 +162,21 @@ static int decode_hsr(struct kvm_vcpu *vcpu, phys_addr_t
> fault_ipa,
> return 0;
> }
>
> +static int handle_io_bus_rw(struct kvm_vcpu *vcpu, gpa_t addr,
> + int len, void *val, bool write)
> +{
> + int idx, ret;
> +
> + idx = srcu_read_lock(&vcpu->kvm->srcu);
> + if (write)
> + ret = kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, addr, len, val);
> + else
> + ret = kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, addr, len, val);
> + srcu_read_unlock(&vcpu->kvm->srcu, idx);
> +
> + return ret;
> +}
> +
> int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run,
>phys_addr_t fault_ipa)
> {
> @@ -200,6 +215,10 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run
> *run,
> if (vgic_handle_mmio(vcpu, run, &mmio))
> return 1;
>
> + if (!handle_io_bus_rw(vcpu, fault_ipa, mmio.len, &mmio.data,
> + mmio.is_write))
> + return 1;
> +
> kvm_prepare_mmio(run, &mmio);
> return 0;
> }
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 8ba85e9..642f57c 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -26,6 +26,7 @@ config KVM
> select KVM_ARM_HOST
> select KVM_ARM_VGIC
> select KVM_ARM_TIMER
> + select HAVE_KVM_EVENTFD
> ---help---
> Support hosting virtualized guest machines.
>
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 32a0961..2e6b827 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm
>
> obj-$(CONFIG_KVM_ARM_HOST) += kvm.o
>
> -kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
> $(KVM)/eventfd.o
> kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o
> kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info
[PATCH v4 0/3] irqfd support for arm/arm64
This patch series enables irqfd on arm and arm64. Irqfd framework enables to inject a virtual IRQ into a guest upon an eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with a kvm_irqfd struct that associates a VM, an eventfd, a virtual IRQ number (aka. the gsi). When an actor signals the eventfd (typically a VFIO platform driver), the kvm irqfd subsystem injects the gsi into the VM. Resamplefd also is supported for level sensitive interrupts, ie. the user can provide another eventfd that is triggered when the completion of the virtual IRQ (gsi) is detected by the GIC. The gsi must correspond to a shared peripheral interrupt (SPI), ie the GIC interrupt ID is gsi + 32. The rationale behind not supporting PPI irqfd injection is that any device using a PPI would be a private-to-the-CPU device (timer for instance), so its state would have to be context-switched along with the VCPU and would require in-kernel wiring anyhow. It is not a relevant use case for irqfds. this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD. No IRQ routing table is used, enabling to remove CONFIG_HAVE_KVM_IRQCHIP can be found at git://git.linaro.org/people/eric.auger/linux.git on branch irqfd_integ_v8 This work was tested with Calxeda Midway xgmac main interrupt with qemu-system-arm and QEMU VFIO platform device. Also irqfd was proven functional on several vhost-net prototypes. v3 -> v4: - rebase on 3.18rc5 - vgic dynamic instantiation brought new challenges: handling of irqfd injection when vgic is not ready - unset of CONFIG_HAVE_KVM_IRQCHIP in a separate patch - add arm64 enable - vgic.c style modifications according to Christoffer comments v2 -> v3: - removal of irq.h from eventfd.c put in a separate patch to increase visibility - properly expose KVM_CAP_IRQFD capability in arm.c - remove CONFIG_HAVE_KVM_IRQCHIP meaningfull only if irq_comm.c is used v1 -> v2: - rebase on 3.17rc1 - move of the dist unlock in process_maintenance - remove of dist lock in __kvm_vgic_sync_hwstate - rewording of the commit message (add resamplefd reference) - remove irq.h Eric Auger (2): KVM: arm/arm64: unset CONFIG_HAVE_KVM_IRQCHIP KVM: arm: add irqfd support Joel Schopp (1): KVM: arm64: add irqfd support Documentation/virtual/kvm/api.txt | 5 ++- arch/arm/include/uapi/asm/kvm.h | 3 ++ arch/arm/kvm/Kconfig | 4 +-- arch/arm/kvm/Makefile | 2 +- arch/arm/kvm/arm.c| 3 ++ arch/arm64/include/uapi/asm/kvm.h | 3 ++ arch/arm64/kvm/Kconfig| 3 +- arch/arm64/kvm/Makefile | 2 +- virt/kvm/arm/vgic.c | 72 --- 9 files changed, 87 insertions(+), 10 deletions(-) -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4 3/3] KVM: arm64: add irqfd support
From: Joel Schopp
This patch enables irqfd for arm64.
Signed-off-by: Joel Schopp
Signed-off-by: Eric Auger
---
[Eric Auger]
- originates from Joel's [RFC PATCH] arm64: KVM: add irqfd support
http://www.spinics.net/lists/kvm-arm/msg10798.html
- isolates modifications really related to irqfd
---
Documentation/virtual/kvm/api.txt | 4 ++--
arch/arm64/include/uapi/asm/kvm.h | 3 +++
arch/arm64/kvm/Kconfig| 2 ++
arch/arm64/kvm/Makefile | 2 +-
4 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/Documentation/virtual/kvm/api.txt
b/Documentation/virtual/kvm/api.txt
index 4deccc0..c76ce04 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2206,7 +2206,7 @@ into the hash PTE second double word).
4.75 KVM_IRQFD
Capability: KVM_CAP_IRQFD
-Architectures: x86 s390 arm
+Architectures: x86 s390 arm arm64
Type: vm ioctl
Parameters: struct kvm_irqfd (in)
Returns: 0 on success, -1 on error
@@ -2232,7 +2232,7 @@ Note that closing the resamplefd is not sufficient to
disable the
irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
-On arm, the gsi must be a shared peripheral interrupt (SPI).
+On arm/arm64, the gsi must be a shared peripheral interrupt (SPI).
This means the corresponding programmed GIC interrupt ID is gsi+32.
4.76 KVM_PPC_ALLOCATE_HTAB
diff --git a/arch/arm64/include/uapi/asm/kvm.h
b/arch/arm64/include/uapi/asm/kvm.h
index 8e38878..1ed4417 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -182,6 +182,9 @@ struct kvm_arch_memory_slot {
/* Highest supported SPI, from VGIC_NR_IRQS */
#define KVM_ARM_IRQ_GIC_MAX127
+/* One single KVM irqchip, ie. the VGIC */
+#define KVM_NR_IRQCHIPS 1
+
/* PSCI interface */
#define KVM_PSCI_FN_BASE 0x95c1ba5e
#define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 279e1a0..09c25c2 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -26,6 +26,7 @@ config KVM
select KVM_ARM_HOST
select KVM_ARM_VGIC
select KVM_ARM_TIMER
+ select HAVE_KVM_EVENTFD
---help---
Support hosting virtualized guest machines.
@@ -50,6 +51,7 @@ config KVM_ARM_MAX_VCPUS
config KVM_ARM_VGIC
bool
depends on KVM_ARM_HOST && OF
+ select HAVE_KVM_IRQFD
---help---
Adds support for a hardware assisted, in-kernel GIC emulation.
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 32a0961..2e6b827 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm
obj-$(CONFIG_KVM_ARM_HOST) += kvm.o
-kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
$(KVM)/eventfd.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4 2/3] KVM: arm: add irqfd support
This patch enables irqfd on arm.
Both irqfd and resamplefd are supported. Injection is implemented
in vgic.c without routing.
This patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
KVM_CAP_IRQFD is now advertised. KVM_CAP_IRQFD_RESAMPLE capability
automatically is advertised as soon as CONFIG_HAVE_KVM_IRQFD is set.
Signed-off-by: Eric Auger
---
v3 -> v4:
- reword commit message
- explain why we unlock the distributor before calling kvm_notify_acked_irq
- rename is_assigned_irq into has_notifier
- change EOI and injection kvm_debug format string
- remove error local variable in kvm_set_irq
- Move HAVE_KVM_IRQCHIP unset in a separate patch
- The rationale behind not supporting PPI irqfd injection is that
any device using a PPI would be a private-to-the-CPU device (timer for
instance), so its state would have to be context-switched along with the
VCPU and would require in-kernel wiring anyhow. It is not a relevant use
case for irqfds.
- handle case were the irqfd injection is attempted before the vgic is ready.
in such a case the notifier, if any, is called immediatly
- use nr_irqs to test spi is within correct range
v2 -> v3:
- removal of irq.h from eventfd.c put in a separate patch to increase
visibility
- properly expose KVM_CAP_IRQFD capability in arm.c
- remove CONFIG_HAVE_KVM_IRQCHIP meaningfull only if irq_comm.c is used
v1 -> v2:
- rebase on 3.17rc1
- move of the dist unlock in process_maintenance
- remove of dist lock in __kvm_vgic_sync_hwstate
- rewording of the commit message (add resamplefd reference)
- remove irq.h
---
Documentation/virtual/kvm/api.txt | 5 ++-
arch/arm/include/uapi/asm/kvm.h | 3 ++
arch/arm/kvm/Kconfig | 2 ++
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/arm.c| 3 ++
virt/kvm/arm/vgic.c | 72 ---
6 files changed, 81 insertions(+), 6 deletions(-)
diff --git a/Documentation/virtual/kvm/api.txt
b/Documentation/virtual/kvm/api.txt
index 7610eaa..4deccc0 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2206,7 +2206,7 @@ into the hash PTE second double word).
4.75 KVM_IRQFD
Capability: KVM_CAP_IRQFD
-Architectures: x86 s390
+Architectures: x86 s390 arm
Type: vm ioctl
Parameters: struct kvm_irqfd (in)
Returns: 0 on success, -1 on error
@@ -2232,6 +2232,9 @@ Note that closing the resamplefd is not sufficient to
disable the
irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
+On arm, the gsi must be a shared peripheral interrupt (SPI).
+This means the corresponding programmed GIC interrupt ID is gsi+32.
+
4.76 KVM_PPC_ALLOCATE_HTAB
Capability: KVM_CAP_PPC_ALLOC_HTAB
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 09ee408..77547bb 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -196,6 +196,9 @@ struct kvm_arch_memory_slot {
/* Highest supported SPI, from VGIC_NR_IRQS */
#define KVM_ARM_IRQ_GIC_MAX127
+/* One single KVM irqchip, ie. the VGIC */
+#define KVM_NR_IRQCHIPS 1
+
/* PSCI interface */
#define KVM_PSCI_FN_BASE 0x95c1ba5e
#define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 9f581b1..e519a40 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -24,6 +24,7 @@ config KVM
select KVM_MMIO
select KVM_ARM_HOST
depends on ARM_VIRT_EXT && ARM_LPAE
+ select HAVE_KVM_EVENTFD
---help---
Support hosting virtualized guest machines. You will also
need to select one or more of the processor modules below.
@@ -55,6 +56,7 @@ config KVM_ARM_MAX_VCPUS
config KVM_ARM_VGIC
bool "KVM support for Virtual GIC"
depends on KVM_ARM_HOST && OF
+ select HAVE_KVM_IRQFD
default y
---help---
Adds support for a hardware assisted, in-kernel GIC emulation.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index f7057ed..859db09 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
KVM := ../../../virt/kvm
-kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
+kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 9e193c8..fb75af2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -172,6 +172,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_IRQCHIP:
r = vgic_present;
break;
+#ifdef CONFIG_HAVE_KVM_IRQFD
+ case
[PATCH v4 1/3] KVM: arm/arm64: unset CONFIG_HAVE_KVM_IRQCHIP
CONFIG_HAVE_KVM_IRQCHIP is needed to support IRQ routing (along with irq_comm.c and irqchip.c usage). This is not the case for arm/arm64 currently. This patch unsets the flag for both arm and arm64. Signed-off-by: Eric Auger --- arch/arm/kvm/Kconfig | 2 -- arch/arm64/kvm/Kconfig | 1 - 2 files changed, 3 deletions(-) diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig index 466bd29..9f581b1 100644 --- a/arch/arm/kvm/Kconfig +++ b/arch/arm/kvm/Kconfig @@ -55,7 +55,6 @@ config KVM_ARM_MAX_VCPUS config KVM_ARM_VGIC bool "KVM support for Virtual GIC" depends on KVM_ARM_HOST && OF - select HAVE_KVM_IRQCHIP default y ---help--- Adds support for a hardware assisted, in-kernel GIC emulation. @@ -63,7 +62,6 @@ config KVM_ARM_VGIC config KVM_ARM_TIMER bool "KVM support for Architected Timers" depends on KVM_ARM_VGIC && ARM_ARCH_TIMER - select HAVE_KVM_IRQCHIP default y ---help--- Adds support for the Architected Timers in virtual machines diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 8ba85e9..279e1a0 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -50,7 +50,6 @@ config KVM_ARM_MAX_VCPUS config KVM_ARM_VGIC bool depends on KVM_ARM_HOST && OF - select HAVE_KVM_IRQCHIP ---help--- Adds support for a hardware assisted, in-kernel GIC emulation. -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 4/4] KVM: arm: vgic: handle irqfd forwarded IRQ injection before vgic readiness
This patch handles the case where irqfd attempts to inject
a forwarded IRQ before the vgic is ready to accept it. We
cannot simply return pretending nothing happened since the IRQ
will never be deactivated by the guest. Indeed, the corresponding
virtual IRQ cannot be injected into the guest and the host
must deactivate it.
Signed-off-by: Eric Auger
---
virt/kvm/arm/vgic.c | 20
1 file changed, 20 insertions(+)
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 21419ac..27394b0 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2672,6 +2672,13 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id,
u32 irq, int level, bool line_status)
{
unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+ struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, 0);
+ int phys_irq = vgic_get_phys_irq(vcpu, spi);
+ bool is_forwarded = (phys_irq >= 0);
+ unsigned long flags;
+ struct irq_desc *desc;
+ struct irq_data *d;
+ struct irq_chip *chip;
kvm_debug("irqfd sets vIRQ %d to %d\n", irq, level);
@@ -2679,6 +2686,19 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id,
if (spi > kvm->arch.vgic.nr_irqs)
return -EINVAL;
return kvm_vgic_inject_irq(kvm, 0, spi, level);
+ }
+
+ if (level && is_forwarded) {
+ /*
+* the virtual IRQ never is injected into the guest
+* it must be manually completed on host side.
+*/
+ desc = irq_to_desc(phys_irq);
+ raw_spin_lock_irqsave(&desc->lock, flags);
+ d = &desc->irq_data;
+ chip = desc->irq_data.chip;
+ chip->irq_eoi(d);
+ raw_spin_unlock_irqrestore(&desc->lock, flags);
} else if (level && kvm_irq_has_notifier(kvm, 0, irq)) {
/*
* any IRQ injected before vgic readiness is
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 1/4] KVM: arm: vgic: fix state machine for forwarded IRQ
Fix multiple injection of level sensitive forwarded IRQs.
With current code, the second injection fails since the state bitmaps
are not reset (process_maintenance is not called anymore).
New implementation follows those principles:
- A forwarded IRQ only can be sampled when it is pending
- when queueing the IRQ (programming the LR), the pending state is removed
as for edge sensitive IRQs
- an injection of a forwarded IRQ is considered always valid since
coming from the HW and level always is 1.
Signed-off-by: Eric Auger
---
v2 -> v3:
- integration in new vgic_can_sample_irq
- remove the pending state when programming the LR
v1 -> v2:
- fix vgic state bypass in vgic_queue_hwirq
Conflicts:
virt/kvm/arm/vgic.c
---
virt/kvm/arm/vgic.c | 16
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index ccb3801..5484e3d 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -360,7 +360,10 @@ static void vgic_cpu_irq_clear(struct kvm_vcpu *vcpu, int
irq)
static bool vgic_can_sample_irq(struct kvm_vcpu *vcpu, int irq)
{
- return vgic_irq_is_edge(vcpu, irq) || !vgic_irq_is_queued(vcpu, irq);
+ bool is_forwarded = (vgic_get_phys_irq(vcpu, irq) >= 0);
+
+ return vgic_irq_is_edge(vcpu, irq) || !vgic_irq_is_queued(vcpu, irq) ||
+ (is_forwarded && vgic_dist_irq_is_pending(vcpu, irq));
}
static u32 mmio_data_read(struct kvm_exit_mmio *mmio, u32 mask)
@@ -1295,6 +1298,7 @@ static bool vgic_queue_irq(struct kvm_vcpu *vcpu, u8
sgi_source_id, int irq)
struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
struct vgic_lr vlr;
int lr;
+ bool is_forwarded = (vgic_get_phys_irq(vcpu, irq) >= 0);
/* Sanitize the input... */
BUG_ON(sgi_source_id & ~7);
@@ -1330,7 +1334,7 @@ static bool vgic_queue_irq(struct kvm_vcpu *vcpu, u8
sgi_source_id, int irq)
vlr.irq = irq;
vlr.source = sgi_source_id;
vlr.state = LR_STATE_PENDING;
- if (!vgic_irq_is_edge(vcpu, irq))
+ if (!vgic_irq_is_edge(vcpu, irq) && !is_forwarded)
vlr.state |= LR_EOI_INT;
vgic_set_lr(vcpu, lr, vlr);
@@ -1371,11 +1375,12 @@ static bool vgic_queue_sgi(struct kvm_vcpu *vcpu, int
irq)
static bool vgic_queue_hwirq(struct kvm_vcpu *vcpu, int irq)
{
+ bool is_forwarded = (vgic_get_phys_irq(vcpu, irq) >= 0);
if (!vgic_can_sample_irq(vcpu, irq))
return true; /* level interrupt, already queued */
if (vgic_queue_irq(vcpu, 0, irq)) {
- if (vgic_irq_is_edge(vcpu, irq)) {
+ if (vgic_irq_is_edge(vcpu, irq) || is_forwarded) {
vgic_dist_irq_clear_pending(vcpu, irq);
vgic_cpu_irq_clear(vcpu, irq);
} else {
@@ -1636,14 +1641,17 @@ static bool vgic_update_irq_pending(struct kvm *kvm,
int cpuid,
int edge_triggered, level_triggered;
int enabled;
bool ret = true;
+ bool is_forwarded;
spin_lock(&dist->lock);
vcpu = kvm_get_vcpu(kvm, cpuid);
+ is_forwarded = (vgic_get_phys_irq(vcpu, irq_num) >= 0);
+
edge_triggered = vgic_irq_is_edge(vcpu, irq_num);
level_triggered = !edge_triggered;
- if (!vgic_validate_injection(vcpu, irq_num, level)) {
+ if (!vgic_validate_injection(vcpu, irq_num, level) && !is_forwarded) {
ret = false;
goto out;
}
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 3/4] KVM: arm: vgic: cleanup forwarded IRQs on destroy
When the VGIC is destroyed it must take care of
- restoring the forwarded IRQs in non forwarded state,
- deactivating the IRQ in case the guest left without doing it
- cleaning nodes of the phys_map rbtree
Signed-off-by: Eric Auger
---
virt/kvm/arm/vgic.c | 36
1 file changed, 36 insertions(+)
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index f592219..21419ac 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -32,6 +32,7 @@
#include
#include
#include
+#include
/*
* How the whole thing works (courtesy of Christoffer Dall):
@@ -102,6 +103,8 @@ static struct vgic_lr vgic_get_lr(const struct kvm_vcpu
*vcpu, int lr);
static void vgic_set_lr(struct kvm_vcpu *vcpu, int lr, struct vgic_lr lr_desc);
static void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
static void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
+static void vgic_clean_irq_phys_map(struct kvm_vcpu *vcpu,
+ struct rb_root *root);
static const struct vgic_ops *vgic_ops;
static const struct vgic_params *vgic;
@@ -1813,6 +1816,36 @@ static struct irq_phys_map *vgic_irq_map_search(struct
kvm_vcpu *vcpu,
return NULL;
}
+static void vgic_clean_irq_phys_map(struct kvm_vcpu *vcpu,
+ struct rb_root *root)
+{
+ unsigned long flags;
+
+ while (1) {
+ struct rb_node *node = rb_first(root);
+ struct irq_phys_map *map;
+ struct irq_desc *desc;
+ struct irq_data *d;
+ struct irq_chip *chip;
+
+ if (!node)
+ break;
+
+ map = container_of(node, struct irq_phys_map, node);
+ desc = irq_to_desc(map->phys_irq);
+
+ raw_spin_lock_irqsave(&desc->lock, flags);
+ d = &desc->irq_data;
+ chip = desc->irq_data.chip;
+ irqd_clr_irq_forwarded(d);
+ chip->irq_eoi(d);
+ raw_spin_unlock_irqrestore(&desc->lock, flags);
+
+ rb_erase(node, root);
+ kfree(map);
+ }
+}
+
int vgic_get_phys_irq(struct kvm_vcpu *vcpu, int virt_irq)
{
struct irq_phys_map *map;
@@ -1855,6 +1888,7 @@ void kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu)
{
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+ vgic_clean_irq_phys_map(vcpu, &vgic_cpu->irq_phys_map);
kfree(vgic_cpu->pending_shared);
kfree(vgic_cpu->vgic_irq_lr_map);
vgic_cpu->pending_shared = NULL;
@@ -1920,6 +1954,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
kvm_for_each_vcpu(i, vcpu, kvm)
kvm_vgic_vcpu_destroy(vcpu);
+ vgic_clean_irq_phys_map(vcpu, &dist->irq_phys_map);
+
vgic_free_bitmap(&dist->irq_enabled);
vgic_free_bitmap(&dist->irq_level);
vgic_free_bitmap(&dist->irq_pending);
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 2/4] KVM: arm: vgic: add forwarded irq rbtree lock
Add a lock related to the rb tree manipulation. The rb tree can be
searched in one thread (irqfd handler for instance) and map/unmap
may happen in another.
Signed-off-by: Eric Auger
---
v2 -> v3:
re-arrange lock sequence in vgic_map_phys_irq
---
include/kvm/arm_vgic.h | 1 +
virt/kvm/arm/vgic.c| 56 --
2 files changed, 42 insertions(+), 15 deletions(-)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index ea31ac6..9b3290b 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -220,6 +220,7 @@ struct vgic_dist {
unsigned long *irq_pending_on_cpu;
struct rb_root irq_phys_map;
+ spinlock_t rb_tree_lock;
#endif
};
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 5484e3d..f592219 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1750,9 +1750,22 @@ static struct rb_root *vgic_get_irq_phys_map(struct
kvm_vcpu *vcpu,
int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct rb_root *root = vgic_get_irq_phys_map(vcpu, virt_irq);
- struct rb_node **new = &root->rb_node, *parent = NULL;
+ struct rb_root *root;
+ struct rb_node **new, *parent = NULL;
struct irq_phys_map *new_map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ root = vgic_get_irq_phys_map(vcpu, virt_irq);
+ new = &root->rb_node;
+
+ new_map = kzalloc(sizeof(*new_map), GFP_KERNEL);
+ if (!new_map)
+ return -ENOMEM;
+
+ new_map->virt_irq = virt_irq;
+ new_map->phys_irq = phys_irq;
+
+ spin_lock(&dist->rb_tree_lock);
/* Boilerplate rb_tree code */
while (*new) {
@@ -1764,19 +1777,16 @@ int vgic_map_phys_irq(struct kvm_vcpu *vcpu, int
virt_irq, int phys_irq)
new = &(*new)->rb_left;
else if (this->virt_irq > virt_irq)
new = &(*new)->rb_right;
- else
+ else {
+ kfree(new_map);
+ spin_unlock(&dist->rb_tree_lock);
return -EEXIST;
+ }
}
- new_map = kzalloc(sizeof(*new_map), GFP_KERNEL);
- if (!new_map)
- return -ENOMEM;
-
- new_map->virt_irq = virt_irq;
- new_map->phys_irq = phys_irq;
-
rb_link_node(&new_map->node, parent, new);
rb_insert_color(&new_map->node, root);
+ spin_unlock(&dist->rb_tree_lock);
return 0;
}
@@ -1805,24 +1815,39 @@ static struct irq_phys_map *vgic_irq_map_search(struct
kvm_vcpu *vcpu,
int vgic_get_phys_irq(struct kvm_vcpu *vcpu, int virt_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+ int ret;
+
+ spin_lock(&dist->rb_tree_lock);
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map)
- return map->phys_irq;
+ ret = map->phys_irq;
+ else
+ ret = -ENOENT;
+
+ spin_unlock(&dist->rb_tree_lock);
+ return ret;
- return -ENOENT;
}
int vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, int virt_irq, int phys_irq)
{
- struct irq_phys_map *map = vgic_irq_map_search(vcpu, virt_irq);
+ struct irq_phys_map *map;
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+ spin_lock(&dist->rb_tree_lock);
+
+ map = vgic_irq_map_search(vcpu, virt_irq);
if (map && map->phys_irq == phys_irq) {
rb_erase(&map->node, vgic_get_irq_phys_map(vcpu, virt_irq));
kfree(map);
+ spin_unlock(&dist->rb_tree_lock);
return 0;
}
-
+ spin_unlock(&dist->rb_tree_lock);
return -ENOENT;
}
@@ -2078,6 +2103,7 @@ int kvm_vgic_create(struct kvm *kvm)
}
spin_lock_init(&kvm->arch.vgic.lock);
+ spin_lock_init(&kvm->arch.vgic.rb_tree_lock);
kvm->arch.vgic.in_kernel = true;
kvm->arch.vgic.vctrl_base = vgic->vctrl_base;
kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 0/4] vgic additions for forwarded irq
This series applies on top of "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) series. It brings some extra functionalities that were requested to be able to inject virtual level sensitive IRQs triggered from VFIO/irqfd. It adds: - a specific handling of forwarded IRQ into the VGIC state machine. - deactivation of physical IRQ and unforwarding on vgic destruction - handling of forwarded IRQ injection before the vgic readiness: this was needed because in a sample qemu/vfio use case, qemu registers forwarded IRQ and set up VFIO signaling before the first vcpu run and hence before vgic readiness. At that time some physical IRQ may hit before the VGIC readiness. This is typically observed with Calxeda xgmac on second QEMU run. - rbtree lock addition. Integrated pieces can be found at ssh://git.linaro.org/people/eric.auger/linux.git on branch irqfd_integ_v8 The first 2 patch files were previously part of [RFC v2 0/9] KVM-VFIO IRQ forward control (https://lkml.org/lkml/2014/9/1/347). Eric Auger (4): KVM: arm: vgic: fix state machine for forwarded IRQ KVM: arm: vgic: add forwarded irq rbtree lock KVM: arm: vgic: cleanup forwarded IRQs on destroy KVM: arm: vgic: handle irqfd forwarded IRQ injection before vgic readiness include/kvm/arm_vgic.h | 1 + virt/kvm/arm/vgic.c| 128 + 2 files changed, 110 insertions(+), 19 deletions(-) -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 2/8] KVM: arm64: Enable the KVM-VFIO device
Used by KVM-enabled VFIO-based device passthrough support in QEMU. Signed-off-by: Joel Schopp Signed-off-by: Eric Auger --- Extracted from [RFC PATCH] arm64: KVM: add irqfd support http://www.spinics.net/lists/kvm-arm/msg10798.html --- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/Makefile | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 09c25c2..2edf926 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -26,6 +26,7 @@ config KVM select KVM_ARM_HOST select KVM_ARM_VGIC select KVM_ARM_TIMER + select KVM_VFIO select HAVE_KVM_EVENTFD ---help--- Support hosting virtualized guest machines. diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 2e6b827..81ed091 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm obj-$(CONFIG_KVM_ARM_HOST) += kvm.o -kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 4/8] KVM: kvm-vfio: User API for IRQ forwarding
This patch adds and document a new KVM_DEV_VFIO_DEVICE group
and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
to set a VFIO device IRQ as forwarded or not forwarded.
the command takes as argument a handle to a new struct named
kvm_arch_forwarded_irq.
Signed-off-by: Eric Auger
---
v2 -> v3:
- rework vfio kvm device documentation
- reword commit message and title
- add subindex in kvm_arch_forwarded_irq to be closer to VFIO API
- forwarding state can only be changed with VFIO IRQ signaling is off
v1 -> v2:
- struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h
to include/uapi/linux/kvm.h
also irq_index renamed into index and guest_irq renamed into gsi
- ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD
---
Documentation/virtual/kvm/devices/vfio.txt | 34 --
include/uapi/linux/kvm.h | 10 +
2 files changed, 38 insertions(+), 6 deletions(-)
diff --git a/Documentation/virtual/kvm/devices/vfio.txt
b/Documentation/virtual/kvm/devices/vfio.txt
index ef51740..f7aff29 100644
--- a/Documentation/virtual/kvm/devices/vfio.txt
+++ b/Documentation/virtual/kvm/devices/vfio.txt
@@ -4,15 +4,24 @@ VFIO virtual device
Device types supported:
KVM_DEV_TYPE_VFIO
-Only one VFIO instance may be created per VM. The created device
-tracks VFIO groups in use by the VM and features of those groups
-important to the correctness and acceleration of the VM. As groups
-are enabled and disabled for use by the VM, KVM should be updated
-about their presence. When registered with KVM, a reference to the
-VFIO-group is held by KVM.
+Only one VFIO instance may be created per VM.
+
+The created device tracks VFIO groups in use by the VM and features
+of those groups important to the correctness and acceleration of
+the VM. As groups are enabled and disabled for use by the VM, KVM
+should be updated about their presence. When registered with KVM,
+a reference to the VFIO-group is held by KVM.
+
+The device also tracks & enable VFIO device forwarded IRQs, if any.
+A physical forwarded IRQ is directly completed by the guest. This
+requires HW support in the interrupt controller which must be able
+to automatically complete the physical IRQ when it detects the guest
+has completed the corresponding virtual IRQ. The modality sometimes
+is named direct EOI.
Groups:
KVM_DEV_VFIO_GROUP
+ KVM_DEV_VFIO_DEVICE
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
@@ -20,3 +29,16 @@ KVM_DEV_VFIO_GROUP attributes:
For each, kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
+
+KVM_DEV_VFIO_DEVICE attributes:
+ KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as forwarded
+ KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as not forwarded
+
+For each, kvm_device_attr.addr points to a kvm_arch_forwarded_irq struct.
+
+The forwarded state can only be changed when the VFIO signaling mechanism
+for this physical IRQ is not set. In other words, forwarding must be
+activated before VFIO_DEVICE_SET_IRQS has been called to trigger the IRQ
+or associate an eventfd to it. Unforwarding can only be called while the
+signaling has been disabled with VFIO_DEVICE_SET_IRQS. If this condition is
+not satisfied, the command returns an -EBUSY.
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6076882..a269a42 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -946,6 +946,9 @@ struct kvm_device_attr {
#define KVM_DEV_VFIO_GROUP1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_DEVICE 2
+#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
+#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
enum kvm_device_type {
KVM_DEV_TYPE_FSL_MPIC_20= 1,
@@ -963,6 +966,13 @@ enum kvm_device_type {
KVM_DEV_TYPE_MAX,
};
+struct kvm_arch_forwarded_irq {
+ __u32 fd; /* file desciptor of the VFIO device */
+ __u32 index; /* VFIO device IRQ index */
+ __u32 subindex; /* VFIO device IRQ subindex */
+ __u32 gsi; /* gsi, ie. virtual IRQ number */
+};
+
/*
* ioctls for VM fds
*/
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 8/8] KVM: arm: kvm-vfio: forwarding control
This patch sets __KVM_HAVE_ARCH_KVM_VFIO_FORWARD and implements
kvm_arch_vfio_set_forward for ARM.
As a result the KVM-VFIO device now allows to forward/unforward a
VFIO device IRQ on ARM.
kvm_arch_vfio_set_forward programs both genirq and the VGIC to control
where the physical IRQ deactivation is initiated.
- forwarded case: deactivation is initiated by the guest; when it
completes the virtual IRQ, the GIC automatically deactivates the
physical IRQ.
- not forwarded case: the physical IRQ deactivation is handled by the host
Signed-off-by: Eric Auger
---
v2 -> v3:
- renaming of kvm_arch_set_fwd_state into kvm_arch_vfio_set_forward
- takes a bool arg instead of kvm_fwd_irq_action enum
- removal of KVM_VFIO_IRQ_CLEANUP
- platform device check now happens here
- more precise errors returned
- irq_eoi handled externally to this patch (VGIC)
- correct enable_irq bug done twice
- reword the commit message
- correct check of platform_bus_type
- use raw_spin_lock_irqsave and check the validity of the handler
---
arch/arm/include/asm/kvm_host.h | 2 +
arch/arm/kvm/Makefile | 2 +-
arch/arm/kvm/kvm_vfio_arm.c | 101
3 files changed, 104 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/kvm/kvm_vfio_arm.c
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index bca5b79..447f90c 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -27,6 +27,8 @@
#include
#include
+#define __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
+
#if defined(CONFIG_KVM_ARM_MAX_VCPUS)
#define KVM_MAX_VCPUS CONFIG_KVM_ARM_MAX_VCPUS
#else
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index ea1fa76..26a5a42 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -19,7 +19,7 @@ kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o
$(KVM)/eventfd.o $(KVM)/vf
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
-obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o
+obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o kvm_vfio_arm.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic.o
obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic-v2.o
obj-$(CONFIG_KVM_ARM_TIMER) += $(KVM)/arm/arch_timer.o
diff --git a/arch/arm/kvm/kvm_vfio_arm.c b/arch/arm/kvm/kvm_vfio_arm.c
new file mode 100644
index 000..af2c501
--- /dev/null
+++ b/arch/arm/kvm/kvm_vfio_arm.c
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2014 Linaro Ltd.
+ * Authors: Eric Auger
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+/**
+ * kvm_arch_vfio_set_forward - Change the forward state of an IRQ
+ *
+ * @fwd_irq: handle to the forward irq struct
+ * @forward: target forwarding state
+ *
+ * If forward is true, programs genirq and VGIC so that physical IRQ
+ * deactivation ownership is transferred to the guest (using GIC HW feature).
+ * When forward is false, standard behavior is restored, ie. host
+ * deactivates the physical IRQ.
+ * returns:
+ * -EINVAL if the vfio device is not a platform device
+ * -ENOENT if the irq could not be identified
+ * -EBUSY if physical IRQ is in progress
+ * -ENOENT if the VGIC has a physical/virtual IRQ mapping that is not
+ * consistent with the request.
+ */
+int kvm_arch_vfio_set_forward(struct kvm_fwd_irq *fwd_irq,
+ bool forward)
+{
+ int hwirq;
+ int ret = -EBUSY;
+ struct irq_desc *desc;
+ struct irq_data *d;
+ struct platform_device *platdev;
+ struct device *dev = kvm_vfio_external_base_device(fwd_irq->vdev);
+ unsigned long flags;
+ /*
+* We don't have to garantee the vcpu handle is non void since the
+* vfio device holds a reference to the kvm struct
+*/
+ struct kvm_vcpu *vcpu = kvm_get_vcpu(fwd_irq->kvm, 0);
+
+ if (dev->bus == &platform_bus_type) {
+ platdev = to_platform_device(dev);
+ hwirq = platform_get_irq(platdev, fwd_irq->index);
+ if (hwirq < 0)
+ return -EINVAL;
+ } else
+ return -ENOENT;
+ desc = irq_to_desc(hwirq);
+
+ /*
+* if VFIO handler is already set, forwarded state cannot be
+* changed anymore
+*/
+ raw_spin_lock_irqsave(&desc->lock, flags);
+ if (desc->action)
+ go
[PATCH v3 6/8] KVM: kvm-vfio: wrapper to VFIO external API device helpers
Provide wrapper functions that allow KVM-VFIO device code to
interact with a vfio device:
- kvm_vfio_device_get_external_user gets a handle to a struct
vfio_device from the vfio device file descriptor and increments
its reference counter,
- kvm_vfio_device_put_external_user decrements the reference counter
to a vfio device,
- kvm_vfio_external_base_device returns a handle to the struct device
of the vfio device.
The KVM-VFIO device uses the VFIO external API device functions.
Signed-off-by: Eric Auger
---
v2 -> v3:
reword the commit message and title
v1 -> v2:
- kvm_vfio_external_get_base_device renamed into
kvm_vfio_external_base_device
- kvm_vfio_external_get_type removed
---
arch/arm/include/asm/kvm_host.h | 5 +
virt/kvm/vfio.c | 45 +
2 files changed, 50 insertions(+)
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 53036e2..bca5b79 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -169,6 +169,11 @@ void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva,
pte_t pte);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
+struct vfio_device;
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep);
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev);
+struct device *kvm_vfio_external_base_device(struct vfio_device *vdev);
+
/* We do not have shadow page tables, hence the empty hooks */
static inline int kvm_age_hva(struct kvm *kvm, unsigned long start,
unsigned long end)
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index 620e37f..6f0cc34 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -60,6 +60,51 @@ static void kvm_vfio_group_put_external_user(struct
vfio_group *vfio_group)
symbol_put(vfio_group_put_external_user);
}
+struct vfio_device *kvm_vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev;
+ struct vfio_device *(*fn)(struct file *);
+
+ fn = symbol_get(vfio_device_get_external_user);
+ if (!fn)
+ return ERR_PTR(-EINVAL);
+
+ vdev = fn(filep);
+
+ symbol_put(vfio_device_get_external_user);
+
+ return vdev;
+}
+
+void kvm_vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ void (*fn)(struct vfio_device *);
+
+ fn = symbol_get(vfio_device_put_external_user);
+ if (!fn)
+ return;
+
+ fn(vdev);
+
+ symbol_put(vfio_device_put_external_user);
+}
+
+struct device *kvm_vfio_external_base_device(struct vfio_device *vdev)
+{
+ struct device *(*fn)(struct vfio_device *);
+ struct device *dev;
+
+ fn = symbol_get(vfio_external_base_device);
+ if (!fn)
+ return NULL;
+
+ dev = fn(vdev);
+
+ symbol_put(vfio_external_base_device);
+
+ return dev;
+}
+
static bool kvm_vfio_group_is_coherent(struct vfio_group *vfio_group)
{
long (*fn)(struct vfio_group *, unsigned long);
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 7/8] KVM: kvm-vfio: generic forwarding control
This patch introduces a new KVM_DEV_VFIO_DEVICE group.
This is a new control channel which enables KVM to cooperate with
viable VFIO devices.
Functions are introduced to check the validity of a VFIO device
file descriptor, increment/decrement the ref counter of the VFIO
device.
The patch introduces 2 attributes for this new device group:
KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
Their purpose is to turn a VFIO device IRQ into a forwarded IRQ and
unset respectively unset the feature.
The VFIO device stores a list of registered forwarded IRQs. The reference
counter of the device is incremented each time a new IRQ is forwarded.
Reference counter is decremented when the IRQ forwarding is unset.
The forwarding programmming is architecture specific, implemented in
kvm_arch_set_fwd_state function. Architecture specific implementation is
enabled when __KVM_HAVE_ARCH_KVM_VFIO_FORWARD is set. When not set those
functions are void.
Signed-off-by: Eric Auger
---
v2 -> v3:
- add API comments in kvm_host.h
- improve the commit message
- create a private kvm_vfio_fwd_irq struct
- fwd_irq_action replaced by a bool and removal of VFIO_IRQ_CLEANUP. This
latter action will be handled in vgic.
- add a vfio_device handle argument to kvm_arch_set_fwd_state. The goal is
to move platform specific stuff in architecture specific code.
- kvm_arch_set_fwd_state renamed into kvm_arch_vfio_set_forward
- increment the ref counter each time we do an IRQ forwarding and decrement
this latter each time one IRQ forward is unset. Simplifies the whole
ref counting.
- simplification of list handling: create, search, removal
v1 -> v2:
- __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
- original patch file separated into 2 parts: generic part moved in vfio.c
and ARM specific part(kvm_arch_set_fwd_state)
---
include/linux/kvm_host.h | 28 ++
virt/kvm/vfio.c | 249 ++-
2 files changed, 274 insertions(+), 3 deletions(-)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ea53b04..0b9659d 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1076,6 +1076,15 @@ struct kvm_device_ops {
unsigned long arg);
};
+/* internal self-contained structure describing a forwarded IRQ */
+struct kvm_fwd_irq {
+ struct kvm *kvm; /* VM to inject the GSI into */
+ struct vfio_device *vdev; /* vfio device the IRQ belongs to */
+ __u32 index; /* VFIO device IRQ index */
+ __u32 subindex; /* VFIO device IRQ subindex */
+ __u32 gsi; /* gsi, ie. virtual IRQ number */
+};
+
void kvm_device_get(struct kvm_device *dev);
void kvm_device_put(struct kvm_device *dev);
struct kvm_device *kvm_device_from_filp(struct file *filp);
@@ -1085,6 +1094,25 @@ void kvm_unregister_device_ops(u32 type);
extern struct kvm_device_ops kvm_mpic_ops;
extern struct kvm_device_ops kvm_xics_ops;
+#ifdef __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
+/**
+ * kvm_arch_vfio_set_forward - changes the forwarded state of an IRQ
+ *
+ * @fwd_irq: handle to the forwarded irq struct
+ * @forward: true means forwarded, false means not forwarded
+ * returns 0 on success, < 0 on failure
+ */
+int kvm_arch_vfio_set_forward(struct kvm_fwd_irq *fwd_irq,
+ bool forward);
+
+#else
+static inline int kvm_arch_vfio_set_forward(struct kvm_fwd_irq *fwd_irq,
+ bool forward)
+{
+ return 0;
+}
+#endif
+
#ifdef CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT
static inline void kvm_vcpu_set_in_spin_loop(struct kvm_vcpu *vcpu, bool val)
diff --git a/virt/kvm/vfio.c b/virt/kvm/vfio.c
index 6f0cc34..af178bb 100644
--- a/virt/kvm/vfio.c
+++ b/virt/kvm/vfio.c
@@ -25,8 +25,16 @@ struct kvm_vfio_group {
struct vfio_group *vfio_group;
};
+/* private linkable kvm_fwd_irq struct */
+struct kvm_vfio_fwd_irq_node {
+ struct list_head link;
+ struct kvm_fwd_irq fwd_irq;
+};
+
struct kvm_vfio {
struct list_head group_list;
+ /* list of registered VFIO forwarded IRQs */
+ struct list_head fwd_node_list;
struct mutex lock;
bool noncoherent;
};
@@ -247,12 +255,239 @@ static int kvm_vfio_set_group(struct kvm_device *dev,
long attr, u64 arg)
return -ENXIO;
}
+/**
+ * kvm_vfio_get_vfio_device - Returns a handle to a vfio-device
+ *
+ * Checks it is a valid vfio device and increments its reference counter
+ * @fd: file descriptor of the vfio platform device
+ */
+static struct vfio_device *kvm_vfio_get_vfio_device(int fd)
+{
+ struct fd f = fdget(fd);
+ struct vfio_device *vdev;
+
+ if (!f.file)
+ return NULL;
+ vdev = kvm_vfio_device_get_external_user(f.file);
+ fdput(f);
+ return vdev;
+}
+
+/**
+ * kvm_vfio_put_vfio_device: decrements the reference counter of the
+ * vfio platform * device
+ *
+ * @vdev: vfio_device handle t
[PATCH v3 5/8] VFIO: External user API device helpers
The VFIO external user API is enriched with 3 new functions that
allows a kernel user external to VFIO to retrieve some information
from a VFIO device.
- vfio_device_get_external_user enables to get a vfio device from
its fd and increments its reference counter
- vfio_device_put_external_user decrements the reference counter
- vfio_external_base_device returns a handle to the struct device
Signed-off-by: Eric Auger
---
v2 -> v3:
- reword the commit message
v1 -> v2:
- vfio_external_get_base_device renamed into vfio_external_base_device
- vfio_external_get_type removed
---
drivers/vfio/vfio.c | 24
include/linux/vfio.h | 3 +++
2 files changed, 27 insertions(+)
diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index 8e84471..282814e 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1401,6 +1401,30 @@ void vfio_group_put_external_user(struct vfio_group
*group)
}
EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
+struct vfio_device *vfio_device_get_external_user(struct file *filep)
+{
+ struct vfio_device *vdev = filep->private_data;
+
+ if (filep->f_op != &vfio_device_fops)
+ return ERR_PTR(-EINVAL);
+
+ vfio_device_get(vdev);
+ return vdev;
+}
+EXPORT_SYMBOL_GPL(vfio_device_get_external_user);
+
+void vfio_device_put_external_user(struct vfio_device *vdev)
+{
+ vfio_device_put(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_device_put_external_user);
+
+struct device *vfio_external_base_device(struct vfio_device *vdev)
+{
+ return vdev->dev;
+}
+EXPORT_SYMBOL_GPL(vfio_external_base_device);
+
int vfio_external_user_iommu_id(struct vfio_group *group)
{
return iommu_group_id(group->iommu_group);
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 2fb2e30..08e33ec 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -99,6 +99,9 @@ extern void vfio_group_put_external_user(struct vfio_group
*group);
extern int vfio_external_user_iommu_id(struct vfio_group *group);
extern long vfio_external_check_extension(struct vfio_group *group,
unsigned long arg);
+extern struct vfio_device *vfio_device_get_external_user(struct file *filep);
+extern void vfio_device_put_external_user(struct vfio_device *vdev);
+extern struct device *vfio_external_base_device(struct vfio_device *vdev);
struct pci_dev;
#ifdef CONFIG_EEH
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 3/8] VFIO: platform: forwarded state tested when selecting IRQ handler
In case the IRQ is forwarded, the VFIO platform IRQ handler does not
need to disable the IRQ anymore.
When setting the IRQ handler we now also test the forwarded state. In
case the IRQ is forwarded we select the edge handler (no automaske).
Signed-off-by: Eric Auger
---
v2 -> v3:
- forwarded state was tested in the handler. Now the forwarded state
is tested before setting the handler. This definitively limits
the dynamics of forwarded state changes but I don't think there is
a use case where we need to be able to change the state at any time.
---
drivers/vfio/platform/vfio_platform_irq.c | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vfio/platform/vfio_platform_irq.c
b/drivers/vfio/platform/vfio_platform_irq.c
index 08d400e..61a2920 100644
--- a/drivers/vfio/platform/vfio_platform_irq.c
+++ b/drivers/vfio/platform/vfio_platform_irq.c
@@ -230,8 +230,13 @@ static int vfio_platform_set_irq_trigger(struct
vfio_platform_device *vdev,
{
struct vfio_platform_irq *irq = &vdev->irqs[index];
irq_handler_t handler;
+ struct irq_data *d;
+ bool is_forwarded;
- if (vdev->irqs[index].flags & VFIO_IRQ_INFO_MASKABLE)
+ d = irq_get_irq_data(irq->hwirq);
+ is_forwarded = irqd_irq_forwarded(d);
+
+ if ((vdev->irqs[index].flags & VFIO_IRQ_INFO_MASKABLE) && !is_forwarded)
handler = vfio_maskable_irq_handler;
else
handler = vfio_irq_handler;
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 1/8] KVM: arm: Enable the KVM-VFIO device
From: Kim Phillips Used by KVM-enabled VFIO-based device passthrough support in QEMU. Signed-off-by: Kim Phillips --- arch/arm/kvm/Kconfig | 1 + arch/arm/kvm/Makefile | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig index e519a40..aace254 100644 --- a/arch/arm/kvm/Kconfig +++ b/arch/arm/kvm/Kconfig @@ -24,6 +24,7 @@ config KVM select KVM_MMIO select KVM_ARM_HOST depends on ARM_VIRT_EXT && ARM_LPAE + select KVM_VFIO select HAVE_KVM_EVENTFD ---help--- Support hosting virtualized guest machines. You will also diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile index 859db09..ea1fa76 100644 --- a/arch/arm/kvm/Makefile +++ b/arch/arm/kvm/Makefile @@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt) AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt) KVM := ../../../virt/kvm -kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o +kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o obj-y += kvm-arm.o init.o interrupts.o obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/8] KVM-VFIO IRQ forward control
This series proposes an integration of "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) in KVM. It enables to transform a VFIO platform driver IRQ into a forwarded IRQ. When a physical IRQ is forwarded (to a guest), the host does not deactivates this latter. Completion ownership is transferred to the guest. When the guest deactivates the associated virtual IRQ, the interrupt controler automatically completes the physical IRQ. Obviously this requires some dedicated HW support in the interrupt controler. The direct benefit is that, for a level sensitive IRQ, it avoids a VM exit on forwarded IRQ completion. When the IRQ is forwarded, the VFIO platform driver does not need to mask the physical IRQ anymore before signaling the eventfd. Indeed genirq lowers the running priority, enabling other physical IRQ to hit except that one. Besides, the injection still is based on irqfd triggering. The only impact on irqfd process is resamplefd is not called anymore on virtual IRQ completion since this latter becomes "transparent". The current integration is based on an extension of the KVM-VFIO device, previously used by KVM to interact with VFIO groups. The patch series now enables KVM to directly interact with a VFIO platform device. The VFIO external API was extended for that purpose. Th KVM-VFIO device can get/put the vfio platform device, check its integrity and type, get the IRQ number associated to an IRQ index. The IRQ forward programming is architecture specific (virtual interrupt controller programming basically). However the whole infrastructure is kept generic. from a user point of view, the functionality is provided through a new KVM-VFIO group named KVM_DEV_VFIO_DEVICE and 2 associated attributes: - KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, - KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The capability can be checked with KVM_HAS_DEVICE_ATTR. Forwarding must be activated before VFIO signaling mechanism is set using VFIO_DEVICE_SET_IRQS and unset while the signaling is disabled. --- This patch series has the following dependencies: - "ARM: Forwarding physical interrupts to a guest VM" (http://lwn.net/Articles/603514/) - [PATCH v9 00/19] VFIO support for platform and AMBA devices on ARM (http://www.spinics.net/lists/kvm-arm/msg11745.html) - [PATCH v2 0/6] vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1 (http://www.spinics.net/lists/kvm-arm/msg11738.html) Integrated pieces can be found at ssh://git.linaro.org/people/eric.auger/linux.git on branch irqfd_integ_v8 This was was tested on Calxeda Midway, assigning the xgmac main IRQ. v2 -> v3: - kvm_fwd_irq_action enum replaced by a bool (KVM_VFIO_IRQ_CLEANUP does not exist anymore) - a new struct local to vfio.c was introduced to wrap kvm_fw_irq and make it linkable: kvm_vfio_fwd_irq_node - kvm_fwd_irq now is self-contained (includes struct vfio_device *) - a single list of kvm_vfio_fwd_irq_irq_node is used instead of having a list of devices and a list of forward irq per device. Having 2 lists brought extra complexity. - the VFIO device ref counter is incremented each time a new IRQ is forwarded. It is not attempted anymore to hold a single reference whatever the number of forwarded IRQs. - subindex added on top of index to be closer to VFIO API - platform device check moved in the arm specific implementation - enable the KVM-VFIO device for arm64 - forwarded state change only can happen while the VFIO IRQ handler is not set; in other words, when the VFIO IRQ signaling is not set. v1 -> v2: - forward control is moved from architecture specific file into generic vfio.c module. only kvm_arch_set_fwd_state remains architecture specific - integrate Kim's patch which enables KVM-VFIO for ARM - fix vgic state bypass in vgic_queue_hwirq - struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h to include/uapi/linux/kvm.h also irq_index renamed into index and guest_irq renamed into gsi - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD - vfio_external_get_base_device renamed into vfio_external_base_device - vfio_external_get_type removed - kvm_vfio_external_get_base_device renamed into kvm_vfio_external_base_device - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD Eric Auger (7): KVM: arm64: Enable the KVM-VFIO device VFIO: platform: forwarded state tested when selecting IRQ handler KVM: kvm-vfio: User API for IRQ forwarding VFIO: External user API device helpers KVM: kvm-vfio: wrapper to VFIO external API device helpers KVM: kvm-vfio: generic forwarding control KVM: arm: kvm-vfio: forwarding control Kim Phillips (1): KVM: arm: Enable the KVM-VFIO device Documentation/virtual/kvm/devices/vfio.txt | 34 +++- arch/arm/include/asm/kvm_host.h| 7 + arch/arm/kvm/Kconfig | 1 + arch/arm/kvm/Makefile | 4 +- arch/arm/kvm/kvm_vfio_arm.c
Re: [PATCH v3 0/8] KVM-VFIO IRQ forward control
On 11/24/2014 09:14 AM, Wu, Feng wrote: > > >> -Original Message----- >> From: Eric Auger [mailto:eric.au...@linaro.org] >> Sent: Monday, November 24, 2014 2:36 AM >> To: eric.au...@st.com; eric.au...@linaro.org; christoffer.d...@linaro.org; >> marc.zyng...@arm.com; linux-arm-ker...@lists.infradead.org; >> kvm...@lists.cs.columbia.edu; kvm@vger.kernel.org; >> alex.william...@redhat.com; joel.sch...@amd.com; >> kim.phill...@freescale.com; pau...@samba.org; g...@kernel.org; >> pbonz...@redhat.com; ag...@suse.de >> Cc: linux-ker...@vger.kernel.org; patc...@linaro.org; will.dea...@arm.com; >> a.mota...@virtualopensystems.com; a.r...@virtualopensystems.com; >> john.li...@huawei.com; ming@canonical.com; Wu, Feng >> Subject: [PATCH v3 0/8] KVM-VFIO IRQ forward control >> >> This series proposes an integration of "ARM: Forwarding physical >> interrupts to a guest VM" (http://lwn.net/Articles/603514/) in >> KVM. >> >> It enables to transform a VFIO platform driver IRQ into a forwarded >> IRQ. >> >> When a physical IRQ is forwarded (to a guest), the host does not >> deactivates this latter. Completion ownership is transferred to the >> guest. When the guest deactivates the associated virtual IRQ, >> the interrupt controler automatically completes the physical IRQ. >> Obviously this requires some dedicated HW support in the interrupt >> controler. >> >> The direct benefit is that, for a level sensitive IRQ, it avoids a >> VM exit on forwarded IRQ completion. >> >> When the IRQ is forwarded, the VFIO platform driver does not need to >> mask the physical IRQ anymore before signaling the eventfd. Indeed >> genirq lowers the running priority, enabling other physical IRQ to hit >> except that one. >> >> Besides, the injection still is based on irqfd triggering. The only >> impact on irqfd process is resamplefd is not called anymore on >> virtual IRQ completion since this latter becomes "transparent". >> >> The current integration is based on an extension of the KVM-VFIO >> device, previously used by KVM to interact with VFIO groups. The >> patch series now enables KVM to directly interact with a VFIO >> platform device. The VFIO external API was extended for that purpose. >> >> Th KVM-VFIO device can get/put the vfio platform device, check its >> integrity and type, get the IRQ number associated to an IRQ index. >> >> The IRQ forward programming is architecture specific (virtual interrupt >> controller programming basically). However the whole infrastructure is >> kept generic. >> >> from a user point of view, the functionality is provided through a >> new KVM-VFIO group named KVM_DEV_VFIO_DEVICE and 2 associated >> attributes: >> - KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, >> - KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. >> >> The capability can be checked with KVM_HAS_DEVICE_ATTR. >> >> Forwarding must be activated before VFIO signaling mechanism is set >> using VFIO_DEVICE_SET_IRQS and unset while the signaling is disabled. >> >> --- >> >> This patch series has the following dependencies: >> - "ARM: Forwarding physical interrupts to a guest VM" >> (http://lwn.net/Articles/603514/) >> - [PATCH v9 00/19] VFIO support for platform and AMBA devices on ARM >> (http://www.spinics.net/lists/kvm-arm/msg11745.html) >> - [PATCH v2 0/6] vfio: type1: support for ARM SMMUS with >> VFIO_IOMMU_TYPE1 >> (http://www.spinics.net/lists/kvm-arm/msg11738.html) >> >> Integrated pieces can be found at >> ssh://git.linaro.org/people/eric.auger/linux.git >> on branch irqfd_integ_v8 >> >> This was was tested on Calxeda Midway, assigning the xgmac main IRQ. >> > > Hi Eric, > > Did you send out the latest QEMU part for this patch set, I notice that v6 of > The QEMU part is sent out, but seems some structure in this new version > has been changed, such as, struct kvm_arch_forwarded_irq (subindex is added > in this version), so a new patchset in QEMU is also needed. Hi Feng, v7 is available at: http://lists.gnu.org/archive/html/qemu-devel/2014-10/msg03804.html. It already illustrates KVM-VFIO device usage. v8 which will indeed integrate subindex addition will be delivered this week. Best Regards Eric > > Thanks, > Feng > > >> v2 -> v3: >> - kvm_fwd_irq_action enum replaced by a bool (KVM_VFIO_IRQ_CLEANUP does >> not >> exist anymore) >> - a new struct local to vfio.c was introduced to wrap kvm_fw_irq and make it >> linkable: kvm_vfio_fwd_irq_node >
Re: [PATCH v4 0/3] irqfd support for arm/arm64
On 11/24/2014 10:47 AM, Christoffer Dall wrote: > On Sun, Nov 23, 2014 at 06:56:57PM +0100, Eric Auger wrote: >> This patch series enables irqfd on arm and arm64. >> >> Irqfd framework enables to inject a virtual IRQ into a guest upon an >> eventfd trigger. User-side uses KVM_IRQFD VM ioctl to provide KVM with >> a kvm_irqfd struct that associates a VM, an eventfd, a virtual IRQ number >> (aka. the gsi). When an actor signals the eventfd (typically a VFIO >> platform driver), the kvm irqfd subsystem injects the gsi into the VM. >> >> Resamplefd also is supported for level sensitive interrupts, ie. the >> user can provide another eventfd that is triggered when the completion >> of the virtual IRQ (gsi) is detected by the GIC. >> >> The gsi must correspond to a shared peripheral interrupt (SPI), ie the >> GIC interrupt ID is gsi + 32. >> >> The rationale behind not supporting PPI irqfd injection is that >> any device using a PPI would be a private-to-the-CPU device (timer for >> instance), so its state would have to be context-switched along with the >> VCPU and would require in-kernel wiring anyhow. It is not a relevant use >> case for irqfds. >> >> this patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD. >> >> No IRQ routing table is used, enabling to remove CONFIG_HAVE_KVM_IRQCHIP >> >> can be found at git://git.linaro.org/people/eric.auger/linux.git >> on branch irqfd_integ_v8 >> >> This work was tested with Calxeda Midway xgmac main interrupt with >> qemu-system-arm and QEMU VFIO platform device. Also irqfd was proven >> functional on several vhost-net prototypes. >> >> v3 -> v4: >> - rebase on 3.18rc5 >> - vgic dynamic instantiation brought new challenges: >> handling of irqfd injection when vgic is not ready >> - unset of CONFIG_HAVE_KVM_IRQCHIP in a separate patch >> - add arm64 enable >> - vgic.c style modifications according to Christoffer comments >> > > There also seems to be a different split of the patches here? Hi Christoffer, yes I added arm64b support and moved CONFIG_HAVE_KVM_IRQCHIP removal in a separate patch. > > We've probably also reached the point where you need to start rebasing > on Andre's GICv3 patches, which I expect will go in first. the patch applies without conflict on Andre's series. BR Eric > > Thanks, > -Christoffer > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 2/3] KVM: arm: add irqfd support
On 11/24/2014 11:00 AM, Christoffer Dall wrote:
> On Sun, Nov 23, 2014 at 06:56:59PM +0100, Eric Auger wrote:
>> This patch enables irqfd on arm.
>>
>> Both irqfd and resamplefd are supported. Injection is implemented
>> in vgic.c without routing.
>>
>> This patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
>>
>> KVM_CAP_IRQFD is now advertised. KVM_CAP_IRQFD_RESAMPLE capability
>> automatically is advertised as soon as CONFIG_HAVE_KVM_IRQFD is set.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v3 -> v4:
>> - reword commit message
>> - explain why we unlock the distributor before calling kvm_notify_acked_irq
>> - rename is_assigned_irq into has_notifier
>> - change EOI and injection kvm_debug format string
>> - remove error local variable in kvm_set_irq
>> - Move HAVE_KVM_IRQCHIP unset in a separate patch
>> - The rationale behind not supporting PPI irqfd injection is that
>> any device using a PPI would be a private-to-the-CPU device (timer for
>> instance), so its state would have to be context-switched along with the
>> VCPU and would require in-kernel wiring anyhow. It is not a relevant use
>> case for irqfds.
>
> this blob could go in the commit message.
OK
>
>> - handle case were the irqfd injection is attempted before the vgic is ready.
>> in such a case the notifier, if any, is called immediatly
>> - use nr_irqs to test spi is within correct range
>>
>> v2 -> v3:
>> - removal of irq.h from eventfd.c put in a separate patch to increase
>> visibility
>> - properly expose KVM_CAP_IRQFD capability in arm.c
>> - remove CONFIG_HAVE_KVM_IRQCHIP meaningfull only if irq_comm.c is used
>>
>> v1 -> v2:
>> - rebase on 3.17rc1
>> - move of the dist unlock in process_maintenance
>> - remove of dist lock in __kvm_vgic_sync_hwstate
>> - rewording of the commit message (add resamplefd reference)
>> - remove irq.h
>> ---
>> Documentation/virtual/kvm/api.txt | 5 ++-
>> arch/arm/include/uapi/asm/kvm.h | 3 ++
>> arch/arm/kvm/Kconfig | 2 ++
>> arch/arm/kvm/Makefile | 2 +-
>> arch/arm/kvm/arm.c| 3 ++
>> virt/kvm/arm/vgic.c | 72
>> ---
>> 6 files changed, 81 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt
>> b/Documentation/virtual/kvm/api.txt
>> index 7610eaa..4deccc0 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2206,7 +2206,7 @@ into the hash PTE second double word).
>> 4.75 KVM_IRQFD
>>
>> Capability: KVM_CAP_IRQFD
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm
>> Type: vm ioctl
>> Parameters: struct kvm_irqfd (in)
>> Returns: 0 on success, -1 on error
>> @@ -2232,6 +2232,9 @@ Note that closing the resamplefd is not sufficient to
>> disable the
>> irqfd. The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>> and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>
>> +On arm, the gsi must be a shared peripheral interrupt (SPI).
>> +This means the corresponding programmed GIC interrupt ID is gsi+32.
>> +
>
> On ARM, the gsi field in the kvm_irqfd struct specifies the Shared
> Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> given by gsi + 32.
OK
>
>> 4.76 KVM_PPC_ALLOCATE_HTAB
>>
>> Capability: KVM_CAP_PPC_ALLOC_HTAB
>> diff --git a/arch/arm/include/uapi/asm/kvm.h
>> b/arch/arm/include/uapi/asm/kvm.h
>> index 09ee408..77547bb 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -196,6 +196,9 @@ struct kvm_arch_memory_slot {
>> /* Highest supported SPI, from VGIC_NR_IRQS */
>> #define KVM_ARM_IRQ_GIC_MAX 127
>>
>> +/* One single KVM irqchip, ie. the VGIC */
>> +#define KVM_NR_IRQCHIPS 1
>> +
>> /* PSCI interface */
>> #define KVM_PSCI_FN_BASE0x95c1ba5e
>> #define KVM_PSCI_FN(n) (KVM_PSCI_FN_BASE + (n))
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 9f581b1..e519a40 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -24,6 +24,7 @@ config KVM
>> select KVM_MMIO
>> select KVM_ARM_HOST
>> depends on ARM_VIRT_EXT && ARM_LPAE
>> +select HAVE_KVM_EVENTFD
>> ---help---
>>Support hosting virtualized guest mach
Re: [RFC 0/4] vgic additions for forwarded irq
On 11/24/2014 11:54 AM, Marc Zyngier wrote: > On 24/11/14 10:50, Christoffer Dall wrote: >> On Sun, Nov 23, 2014 at 07:12:49PM +0100, Eric Auger wrote: >>> This series applies on top of "ARM: Forwarding physical >>> interrupts to a guest VM" (http://lwn.net/Articles/603514/) >>> series. >> >> Marc and Eric, >> >> Does it make sense to review and look at these patches given the current >> state of the forwarding patches, or should we wait until Marc respins >> that series? > > I'm still in the process of respining all of this (we've had a few > iterations with tglx, but I got sidetracked with the ITS/v2m/MSI side of > things). Hi Marc, Christoffer, for your info, I integrated with kvm-arm64/irq-forward branch prior to ITS integration, not with the RFC. Best Regards Eric > > I'll get back onto this this week. Hopefully. > > Thanks, > > M. > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] kvm: x86: move ioapic.c and irq_comm.c back to arch/x86/
On 11/20/2014 02:42 PM, Paolo Bonzini wrote:
> ia64 does not need them anymore.
>
> Signed-off-by: Paolo Bonzini
> ---
> arch/x86/include/asm/kvm_host.h | 16
> arch/x86/kvm/Makefile | 5 ++---
> {virt => arch/x86}/kvm/ioapic.c | 0
> {virt => arch/x86}/kvm/ioapic.h | 1 -
> {virt => arch/x86}/kvm/irq_comm.c | 4 ++--
> arch/x86/kvm/x86.c| 1 +
> include/linux/kvm_host.h | 22 --
> virt/kvm/eventfd.c| 7 ---
> virt/kvm/kvm_main.c | 3 ---
> 9 files changed, 29 insertions(+), 30 deletions(-)
> rename {virt => arch/x86}/kvm/ioapic.c (100%)
> rename {virt => arch/x86}/kvm/ioapic.h (98%)
> rename {virt => arch/x86}/kvm/irq_comm.c (98%)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 769db36a3001..76ff3e2d8fd2 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -603,6 +603,9 @@ struct kvm_arch {
>
> struct kvm_xen_hvm_config xen_hvm_config;
>
> + /* reads protected by irq_srcu, writes by irq_lock */
> + struct hlist_head mask_notifier_list;
> +
> /* fields used by HYPER-V emulation */
> u64 hv_guest_os_id;
> u64 hv_hypercall;
> @@ -819,6 +822,19 @@ int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa,
> const void *val, int bytes);
> u8 kvm_get_guest_memory_type(struct kvm_vcpu *vcpu, gfn_t gfn);
>
> +struct kvm_irq_mask_notifier {
> + void (*func)(struct kvm_irq_mask_notifier *kimn, bool masked);
> + int irq;
> + struct hlist_node link;
> +};
> +
> +void kvm_register_irq_mask_notifier(struct kvm *kvm, int irq,
> + struct kvm_irq_mask_notifier *kimn);
> +void kvm_unregister_irq_mask_notifier(struct kvm *kvm, int irq,
> + struct kvm_irq_mask_notifier *kimn);
> +void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin,
> + bool mask);
> +
> extern bool tdp_enabled;
>
> u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu);
> diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
> index 25d22b2d6509..ee1cd92b03be 100644
> --- a/arch/x86/kvm/Makefile
> +++ b/arch/x86/kvm/Makefile
> @@ -7,14 +7,13 @@ CFLAGS_vmx.o := -I.
>
> KVM := ../../../virt/kvm
>
> -kvm-y+= $(KVM)/kvm_main.o $(KVM)/ioapic.o \
> - $(KVM)/coalesced_mmio.o $(KVM)/irq_comm.o \
> +kvm-y+= $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o \
> $(KVM)/eventfd.o $(KVM)/irqchip.o $(KVM)/vfio.o
> kvm-$(CONFIG_KVM_DEVICE_ASSIGNMENT) += $(KVM)/assigned-dev.o $(KVM)/iommu.o
> kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o
>
> kvm-y+= x86.o mmu.o emulate.o i8259.o irq.o lapic.o \
> -i8254.o cpuid.o pmu.o
> +i8254.o ioapic.o irq_comm.o cpuid.o pmu.o
> kvm-intel-y += vmx.o
> kvm-amd-y+= svm.o
>
> diff --git a/virt/kvm/ioapic.c b/arch/x86/kvm/ioapic.c
> similarity index 100%
> rename from virt/kvm/ioapic.c
> rename to arch/x86/kvm/ioapic.c
> diff --git a/virt/kvm/ioapic.h b/arch/x86/kvm/ioapic.h
> similarity index 98%
> rename from virt/kvm/ioapic.h
> rename to arch/x86/kvm/ioapic.h
> index dc3baa3a538f..deac8d509f2a 100644
> --- a/virt/kvm/ioapic.h
> +++ b/arch/x86/kvm/ioapic.h
> @@ -96,7 +96,6 @@ int kvm_irq_delivery_to_apic(struct kvm *kvm, struct
> kvm_lapic *src,
> struct kvm_lapic_irq *irq, unsigned long *dest_map);
> int kvm_get_ioapic(struct kvm *kvm, struct kvm_ioapic_state *state);
> int kvm_set_ioapic(struct kvm *kvm, struct kvm_ioapic_state *state);
> -void kvm_vcpu_request_scan_ioapic(struct kvm *kvm);
> void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap,
> u32 *tmr);
>
> diff --git a/virt/kvm/irq_comm.c b/arch/x86/kvm/irq_comm.c
> similarity index 98%
> rename from virt/kvm/irq_comm.c
> rename to arch/x86/kvm/irq_comm.c
> index 1345bde064f5..e9c135b639aa 100644
> --- a/virt/kvm/irq_comm.c
> +++ b/arch/x86/kvm/irq_comm.c
> @@ -234,7 +234,7 @@ void kvm_register_irq_mask_notifier(struct kvm *kvm, int
> irq,
> {
> mutex_lock(&kvm->irq_lock);
> kimn->irq = irq;
> - hlist_add_head_rcu(&kimn->link, &kvm->mask_notifier_list);
> + hlist_add_head_rcu(&kimn->link, &kvm->arch.mask_notifier_list);
> mutex_unlock(&kvm->irq_lock);
> }
>
> @@ -256,7 +256,7 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned
> irqchip, unsigned pin,
> idx = srcu_read_lock(&kvm->irq_srcu);
> gsi = kvm_irq_map_chip_pin(kvm, irqchip, pin);
> if (gsi != -1)
> - hlist_for_each_entry_rcu(kimn, &kvm->mask_notifier_list, link)
> + hlist_for_each_entry_rcu(kimn, &kvm->arch.mask_notifier_list,
> link)
> if (kimn->irq == gsi)
>
Re: [PATCH v4 2/3] KVM: arm: add irqfd support
On 11/25/2014 11:19 AM, Christoffer Dall wrote:
> [Re-adding cc list]
>
> On Mon, Nov 24, 2014 at 05:42:30PM +0100, Eric Auger wrote:
>> On 11/24/2014 04:47 PM, Christoffer Dall wrote:
>>> On Mon, Nov 24, 2014 at 12:02 PM, Eric Auger wrote:
>>>> On 11/24/2014 11:00 AM, Christoffer Dall wrote:
>>>>> On Sun, Nov 23, 2014 at 06:56:59PM +0100, Eric Auger wrote:
>>>>>> This patch enables irqfd on arm.
>>>>>>
>>>>>> Both irqfd and resamplefd are supported. Injection is implemented
>>>>>> in vgic.c without routing.
>>>>>>
>>>>>> This patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
>>>>>>
>>>>>> KVM_CAP_IRQFD is now advertised. KVM_CAP_IRQFD_RESAMPLE capability
>>>>>> automatically is advertised as soon as CONFIG_HAVE_KVM_IRQFD is set.
>>>>>>
>>>>>> Signed-off-by: Eric Auger
>>>>>>
>>>>>> ---
>
> [...]
>
>>>>>> +int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>>>>>> +u32 irq, int level, bool line_status)
>>>>>> +{
>>>>>> +unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>>>>>> +
>>>>>> +kvm_debug("irqfd sets vIRQ %d to %d\n", irq, level);
>>>>>> +
>>>>>> +if (likely(vgic_initialized(kvm))) {
>>>>>> +if (spi > kvm->arch.vgic.nr_irqs)
>>>>>> +return -EINVAL;
>>>>>> +return kvm_vgic_inject_irq(kvm, 0, spi, level);
>>>>>> +} else if (level && kvm_irq_has_notifier(kvm, 0, irq)) {
>>>>>> +/*
>>>>>> + * any IRQ injected before vgic readiness is
>>>>>> + * ignored and the notifier, if any, is called
>>>>>> + * immediately as if the virtual IRQ were completed
>>>>>> + * by the guest
>>>>>> + */
>>>>>> +kvm_notify_acked_irq(kvm, 0, irq);
>>>>>> +return -EAGAIN;
>>>>>
>>>>> This looks fishy, I don't fully understand which case you're catering
>>>>> towards here (the comment is hard to understand), but my gut feeling is
>>>>> that you should back out (probably with an error) if the vgic is not
>>>>> initialized when calling this function. Setting up the routing in the
>>>>> first place should probably error out if the vgic is not available.
>>>> The issue is related to integration with QEMU and VFIO.
>>>> Currently VFIO signaling (we tell VFIO to signal the eventfd on a
>>>> peculiar physical IRQ) and irqfd setup (we tell KVM/IRQFD to handle
>>>> previous eventfd when triggered and inject a GSI) are done by QEMU
>>>> *before* the first vcpu run. so before VGIC is ready.
>>>>
>>>> Both are done in a so called QEMU machine init done notifier. It could
>>>> be done in a QEMU reset notifier but I guess the problem would be the
>>>> same. and I think the same at which QEMU initializes it is correct.
>>>>
>>>> As soon as both VFIO signaling and irqfd are setup, virtual IRQ are
>>>> likely to be injected and this is what happens on some circumstances.
>>>> This happens on the 2d QEMU run after killing the 1st QEMU session. For
>>>> some reason I guess the HW device - in my case the xgmac - was released
>>>> in such a way the IRQ wire was not reset. So as soon as VFIO driver
>>>> calls request_irq, the IRQ hits.
>>>>
>>>> I tried to change that this xgmac driver behavior but I must acknowledge
>>>> that for the time beeing I failed. I will continue investigating that.
>>>>
>>>> Indeed I might be fixing the issue at the wrong place. But anyway this
>>>> relies on the fact the assigned device driver toggled down the IRQ
>>>> properly when releasing. At the time the signaling is setup we have not
>>>> entered yet any driver reset code.
>>>>
>>> I see, it's because of the quirky way we initialize the vgic at time
>>> of running the first CPU, so user space can't really hook into the
>>> sweet spot after initializing the vgic but just before actually
>>> running guest code.
>>
>> Yes currently irqfd generic code has no way to te
Re: [PATCH v4 2/3] KVM: arm: add irqfd support
On 11/25/2014 02:12 PM, Eric Auger wrote:
> On 11/25/2014 11:19 AM, Christoffer Dall wrote:
>> [Re-adding cc list]
>>
>> On Mon, Nov 24, 2014 at 05:42:30PM +0100, Eric Auger wrote:
>>> On 11/24/2014 04:47 PM, Christoffer Dall wrote:
>>>> On Mon, Nov 24, 2014 at 12:02 PM, Eric Auger wrote:
>>>>> On 11/24/2014 11:00 AM, Christoffer Dall wrote:
>>>>>> On Sun, Nov 23, 2014 at 06:56:59PM +0100, Eric Auger wrote:
>>>>>>> This patch enables irqfd on arm.
>>>>>>>
>>>>>>> Both irqfd and resamplefd are supported. Injection is implemented
>>>>>>> in vgic.c without routing.
>>>>>>>
>>>>>>> This patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
>>>>>>>
>>>>>>> KVM_CAP_IRQFD is now advertised. KVM_CAP_IRQFD_RESAMPLE capability
>>>>>>> automatically is advertised as soon as CONFIG_HAVE_KVM_IRQFD is set.
>>>>>>>
>>>>>>> Signed-off-by: Eric Auger
>>>>>>>
>>>>>>> ---
>>
>> [...]
>>
>>>>>>> +int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>>>>>>> +u32 irq, int level, bool line_status)
>>>>>>> +{
>>>>>>> +unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>>>>>>> +
>>>>>>> +kvm_debug("irqfd sets vIRQ %d to %d\n", irq, level);
>>>>>>> +
>>>>>>> +if (likely(vgic_initialized(kvm))) {
>>>>>>> +if (spi > kvm->arch.vgic.nr_irqs)
>>>>>>> +return -EINVAL;
>>>>>>> +return kvm_vgic_inject_irq(kvm, 0, spi, level);
>>>>>>> +} else if (level && kvm_irq_has_notifier(kvm, 0, irq)) {
>>>>>>> +/*
>>>>>>> + * any IRQ injected before vgic readiness is
>>>>>>> + * ignored and the notifier, if any, is called
>>>>>>> + * immediately as if the virtual IRQ were completed
>>>>>>> + * by the guest
>>>>>>> + */
>>>>>>> +kvm_notify_acked_irq(kvm, 0, irq);
>>>>>>> +return -EAGAIN;
>>>>>>
>>>>>> This looks fishy, I don't fully understand which case you're catering
>>>>>> towards here (the comment is hard to understand), but my gut feeling is
>>>>>> that you should back out (probably with an error) if the vgic is not
>>>>>> initialized when calling this function. Setting up the routing in the
>>>>>> first place should probably error out if the vgic is not available.
>>>>> The issue is related to integration with QEMU and VFIO.
>>>>> Currently VFIO signaling (we tell VFIO to signal the eventfd on a
>>>>> peculiar physical IRQ) and irqfd setup (we tell KVM/IRQFD to handle
>>>>> previous eventfd when triggered and inject a GSI) are done by QEMU
>>>>> *before* the first vcpu run. so before VGIC is ready.
>>>>>
>>>>> Both are done in a so called QEMU machine init done notifier. It could
>>>>> be done in a QEMU reset notifier but I guess the problem would be the
>>>>> same. and I think the same at which QEMU initializes it is correct.
>>>>>
>>>>> As soon as both VFIO signaling and irqfd are setup, virtual IRQ are
>>>>> likely to be injected and this is what happens on some circumstances.
>>>>> This happens on the 2d QEMU run after killing the 1st QEMU session. For
>>>>> some reason I guess the HW device - in my case the xgmac - was released
>>>>> in such a way the IRQ wire was not reset. So as soon as VFIO driver
>>>>> calls request_irq, the IRQ hits.
>>>>>
>>>>> I tried to change that this xgmac driver behavior but I must acknowledge
>>>>> that for the time beeing I failed. I will continue investigating that.
>>>>>
>>>>> Indeed I might be fixing the issue at the wrong place. But anyway this
>>>>> relies on the fact the assigned device driver toggled down the IRQ
>>>>> properly when releasing. At the time the signaling is setup we have not
>>>>> entered yet any driver reset code.
>>>>>
>>>
Re: [PATCH v3 7/8] KVM: kvm-vfio: generic forwarding control
On 11/25/2014 05:33 AM, Wu, Feng wrote:
>
>
>> -Original Message-----
>> From: Eric Auger [mailto:eric.au...@linaro.org]
>> Sent: Monday, November 24, 2014 2:36 AM
>> To: eric.au...@st.com; eric.au...@linaro.org; christoffer.d...@linaro.org;
>> marc.zyng...@arm.com; linux-arm-ker...@lists.infradead.org;
>> kvm...@lists.cs.columbia.edu; kvm@vger.kernel.org;
>> alex.william...@redhat.com; joel.sch...@amd.com;
>> kim.phill...@freescale.com; pau...@samba.org; g...@kernel.org;
>> pbonz...@redhat.com; ag...@suse.de
>> Cc: linux-ker...@vger.kernel.org; patc...@linaro.org; will.dea...@arm.com;
>> a.mota...@virtualopensystems.com; a.r...@virtualopensystems.com;
>> john.li...@huawei.com; ming@canonical.com; Wu, Feng
>> Subject: [PATCH v3 7/8] KVM: kvm-vfio: generic forwarding control
>>
>> This patch introduces a new KVM_DEV_VFIO_DEVICE group.
>>
>> This is a new control channel which enables KVM to cooperate with
>> viable VFIO devices.
>>
>> Functions are introduced to check the validity of a VFIO device
>> file descriptor, increment/decrement the ref counter of the VFIO
>> device.
>>
>> The patch introduces 2 attributes for this new device group:
>> KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
>> Their purpose is to turn a VFIO device IRQ into a forwarded IRQ and
>> unset respectively unset the feature.
>>
>> The VFIO device stores a list of registered forwarded IRQs. The reference
>> counter of the device is incremented each time a new IRQ is forwarded.
>> Reference counter is decremented when the IRQ forwarding is unset.
>>
>> The forwarding programmming is architecture specific, implemented in
>> kvm_arch_set_fwd_state function. Architecture specific implementation is
>> enabled when __KVM_HAVE_ARCH_KVM_VFIO_FORWARD is set. When not
>> set those
>> functions are void.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v2 -> v3:
>> - add API comments in kvm_host.h
>> - improve the commit message
>> - create a private kvm_vfio_fwd_irq struct
>> - fwd_irq_action replaced by a bool and removal of VFIO_IRQ_CLEANUP. This
>> latter action will be handled in vgic.
>> - add a vfio_device handle argument to kvm_arch_set_fwd_state. The goal is
>> to move platform specific stuff in architecture specific code.
>> - kvm_arch_set_fwd_state renamed into kvm_arch_vfio_set_forward
>> - increment the ref counter each time we do an IRQ forwarding and decrement
>> this latter each time one IRQ forward is unset. Simplifies the whole
>> ref counting.
>> - simplification of list handling: create, search, removal
>>
>> v1 -> v2:
>> - __KVM_HAVE_ARCH_KVM_VFIO renamed into
>> __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> - original patch file separated into 2 parts: generic part moved in vfio.c
>> and ARM specific part(kvm_arch_set_fwd_state)
>> ---
>> include/linux/kvm_host.h | 28 ++
>> virt/kvm/vfio.c | 249
>> ++-
>> 2 files changed, 274 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index ea53b04..0b9659d 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1076,6 +1076,15 @@ struct kvm_device_ops {
>>unsigned long arg);
>> };
>>
>> +/* internal self-contained structure describing a forwarded IRQ */
>> +struct kvm_fwd_irq {
>> +struct kvm *kvm; /* VM to inject the GSI into */
>> +struct vfio_device *vdev; /* vfio device the IRQ belongs to */
>> +__u32 index; /* VFIO device IRQ index */
>> +__u32 subindex; /* VFIO device IRQ subindex */
>> +__u32 gsi; /* gsi, ie. virtual IRQ number */
>> +};
>> +
>> void kvm_device_get(struct kvm_device *dev);
>> void kvm_device_put(struct kvm_device *dev);
>> struct kvm_device *kvm_device_from_filp(struct file *filp);
>> @@ -1085,6 +1094,25 @@ void kvm_unregister_device_ops(u32 type);
>> extern struct kvm_device_ops kvm_mpic_ops;
>> extern struct kvm_device_ops kvm_xics_ops;
>>
>> +#ifdef __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> +/**
>> + * kvm_arch_vfio_set_forward - changes the forwarded state of an IRQ
>> + *
>> + * @fwd_irq: handle to the forwarded irq struct
>> + * @forward: true means forwarded, false means not forwarded
>> + * returns 0 on success, < 0 on failure
>> + */
>> +int kvm_arch_vfio_set_forward(struct kvm_fwd_irq *fwd_irq,
>> +
Re: [RFC PATCH v2 1/2] KVM: kvm-vfio: User API for VT-d Posted-Interrupts
On 11/25/2014 01:23 PM, Feng Wu wrote:
> This patch adds and documents a new attribute
> KVM_DEV_VFIO_DEVICE_POSTING_IRQ in KVM_DEV_VFIO_DEVICE group.
> This new attribute is used for VT-d Posted-Interrupts.
>
> When guest OS changes the interrupt configuration for an
> assigned device, such as, MSI/MSIx data/address fields,
> QEMU will use this IRQ attribute to tell KVM to update the
> related IRTE according the VT-d Posted-Interrrupts Specification,
> such as, the guest vector should be updated in the related IRTE.
>
> Signed-off-by: Feng Wu
> ---
> Documentation/virtual/kvm/devices/vfio.txt |9 +
> include/uapi/linux/kvm.h | 10 ++
> 2 files changed, 19 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
> b/Documentation/virtual/kvm/devices/vfio.txt
> index f7aff29..39dee86 100644
> --- a/Documentation/virtual/kvm/devices/vfio.txt
> +++ b/Documentation/virtual/kvm/devices/vfio.txt
> @@ -42,3 +42,12 @@ activated before VFIO_DEVICE_SET_IRQS has been called to
> trigger the IRQ
> or associate an eventfd to it. Unforwarding can only be called while the
> signaling has been disabled with VFIO_DEVICE_SET_IRQS. If this condition is
> not satisfied, the command returns an -EBUSY.
> +
> + KVM_DEV_VFIO_DEVICE_POSTING_IRQ: Use posted interrtups mechanism to post
> + the IRQ to guests.
> +For this attribute, kvm_device_attr.addr points to a kvm_posted_intr struct.
> +
> +When guest OS changes the interrupt configuration for an assigned device,
> +such as, MSI/MSIx data/address fields, QEMU will use this IRQ attribute
> +to tell KVM to update the related IRTE according the VT-d Posted-Interrrupts
> +Specification, such as, the guest vector should be updated in the related
> IRTE.
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index a269a42..e5f86ad 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -949,6 +949,7 @@ struct kvm_device_attr {
> #define KVM_DEV_VFIO_DEVICE 2
> #define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ1
> #define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ 2
> +#define KVM_DEV_VFIO_DEVICE_POSTING_IRQ3
>
> enum kvm_device_type {
> KVM_DEV_TYPE_FSL_MPIC_20= 1,
> @@ -973,6 +974,15 @@ struct kvm_arch_forwarded_irq {
> __u32 gsi; /* gsi, ie. virtual IRQ number */
> };
>
> +struct kvm_posted_intr {
> + __u32 argsz;
> + __u32 fd; /* file descriptor of the VFIO device */
> + __u32 index; /* VFIO device IRQ index */
> + __u32 start;
> + __u32 count;
> + int virq[0];/* gsi, ie. virtual IRQ number */
> +};
Hi Feng,
This struct could be used by arm code too. If Alex agrees I could use
that one instead. We just need to find a common sensible name
Best Regards
Eric
> +
> /*
> * ioctls for VM fds
> */
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 7/8] KVM: kvm-vfio: generic forwarding control
On 11/24/2014 09:56 PM, Alex Williamson wrote:
> On Sun, 2014-11-23 at 19:35 +0100, Eric Auger wrote:
>> This patch introduces a new KVM_DEV_VFIO_DEVICE group.
>>
>> This is a new control channel which enables KVM to cooperate with
>> viable VFIO devices.
>>
>> Functions are introduced to check the validity of a VFIO device
>> file descriptor, increment/decrement the ref counter of the VFIO
>> device.
>>
>> The patch introduces 2 attributes for this new device group:
>> KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ.
>> Their purpose is to turn a VFIO device IRQ into a forwarded IRQ and
>> unset respectively unset the feature.
>>
>> The VFIO device stores a list of registered forwarded IRQs. The reference
>> counter of the device is incremented each time a new IRQ is forwarded.
>> Reference counter is decremented when the IRQ forwarding is unset.
>>
>> The forwarding programmming is architecture specific, implemented in
>> kvm_arch_set_fwd_state function. Architecture specific implementation is
>> enabled when __KVM_HAVE_ARCH_KVM_VFIO_FORWARD is set. When not set those
>> functions are void.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v2 -> v3:
>> - add API comments in kvm_host.h
>> - improve the commit message
>> - create a private kvm_vfio_fwd_irq struct
>> - fwd_irq_action replaced by a bool and removal of VFIO_IRQ_CLEANUP. This
>> latter action will be handled in vgic.
>> - add a vfio_device handle argument to kvm_arch_set_fwd_state. The goal is
>> to move platform specific stuff in architecture specific code.
>> - kvm_arch_set_fwd_state renamed into kvm_arch_vfio_set_forward
>> - increment the ref counter each time we do an IRQ forwarding and decrement
>> this latter each time one IRQ forward is unset. Simplifies the whole
>> ref counting.
>> - simplification of list handling: create, search, removal
>>
>> v1 -> v2:
>> - __KVM_HAVE_ARCH_KVM_VFIO renamed into __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> - original patch file separated into 2 parts: generic part moved in vfio.c
>> and ARM specific part(kvm_arch_set_fwd_state)
>> ---
>> include/linux/kvm_host.h | 28 ++
>> virt/kvm/vfio.c | 249
>> ++-
>> 2 files changed, 274 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index ea53b04..0b9659d 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1076,6 +1076,15 @@ struct kvm_device_ops {
>>unsigned long arg);
>> };
>>
>> +/* internal self-contained structure describing a forwarded IRQ */
>> +struct kvm_fwd_irq {
>> +struct kvm *kvm; /* VM to inject the GSI into */
>> +struct vfio_device *vdev; /* vfio device the IRQ belongs to */
>> +__u32 index; /* VFIO device IRQ index */
>> +__u32 subindex; /* VFIO device IRQ subindex */
>> +__u32 gsi; /* gsi, ie. virtual IRQ number */
>> +};
>> +
>> void kvm_device_get(struct kvm_device *dev);
>> void kvm_device_put(struct kvm_device *dev);
>> struct kvm_device *kvm_device_from_filp(struct file *filp);
>> @@ -1085,6 +1094,25 @@ void kvm_unregister_device_ops(u32 type);
>> extern struct kvm_device_ops kvm_mpic_ops;
>> extern struct kvm_device_ops kvm_xics_ops;
>>
>> +#ifdef __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
>> +/**
>> + * kvm_arch_vfio_set_forward - changes the forwarded state of an IRQ
>> + *
>> + * @fwd_irq: handle to the forwarded irq struct
>> + * @forward: true means forwarded, false means not forwarded
>> + * returns 0 on success, < 0 on failure
>> + */
>> +int kvm_arch_vfio_set_forward(struct kvm_fwd_irq *fwd_irq,
>> + bool forward);
>
> We could add a struct device* to the args list or into struct
> kvm_fwd_irq so that arch code doesn't need to touch the vdev. arch code
> has no business dealing with references to the vfio_device.
Hi Alex,
Currently It can't put struct device* into the kvm_fwd_irq struct since
I need to release the vfio_device with
vfio_device_put_external_user(struct vfio_device *vdev)
typically in kvm_vfio_clean_fwd_irq. So I need to store the pointers to
the vfio_device somewhere.
I see 2 solutions: change the proto of
vfio_device_put_external_user(struct vfio_device *vdev) and pass a
struct device* (??)
or change the proto of kvm_arch_vfio_set_forward into
kvm_arch_vfio_set_forward(struct kvm *kvm, struct device *dev, int
index, [int subindex], int gsi,
Re: [PATCH v4 2/3] KVM: arm: add irqfd support
On 11/26/2014 12:31 PM, Christoffer Dall wrote:
> On Tue, Nov 25, 2014 at 02:12:31PM +0100, Eric Auger wrote:
>> On 11/25/2014 11:19 AM, Christoffer Dall wrote:
>>> [Re-adding cc list]
>>>
>>> On Mon, Nov 24, 2014 at 05:42:30PM +0100, Eric Auger wrote:
>>>> On 11/24/2014 04:47 PM, Christoffer Dall wrote:
>>>>> On Mon, Nov 24, 2014 at 12:02 PM, Eric Auger
>>>>> wrote:
>>>>>> On 11/24/2014 11:00 AM, Christoffer Dall wrote:
>>>>>>> On Sun, Nov 23, 2014 at 06:56:59PM +0100, Eric Auger wrote:
>>>>>>>> This patch enables irqfd on arm.
>>>>>>>>
>>>>>>>> Both irqfd and resamplefd are supported. Injection is implemented
>>>>>>>> in vgic.c without routing.
>>>>>>>>
>>>>>>>> This patch enables CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQFD.
>>>>>>>>
>>>>>>>> KVM_CAP_IRQFD is now advertised. KVM_CAP_IRQFD_RESAMPLE capability
>>>>>>>> automatically is advertised as soon as CONFIG_HAVE_KVM_IRQFD is set.
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger
>>>>>>>>
>>>>>>>> ---
>>>
>>> [...]
>>>
>>>>>>>> +int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>>>>>>>> +u32 irq, int level, bool line_status)
>>>>>>>> +{
>>>>>>>> +unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>>>>>>>> +
>>>>>>>> +kvm_debug("irqfd sets vIRQ %d to %d\n", irq, level);
>>>>>>>> +
>>>>>>>> +if (likely(vgic_initialized(kvm))) {
>>>>>>>> +if (spi > kvm->arch.vgic.nr_irqs)
>>>>>>>> +return -EINVAL;
>>>>>>>> +return kvm_vgic_inject_irq(kvm, 0, spi, level);
>>>>>>>> +} else if (level && kvm_irq_has_notifier(kvm, 0, irq)) {
>>>>>>>> +/*
>>>>>>>> + * any IRQ injected before vgic readiness is
>>>>>>>> + * ignored and the notifier, if any, is called
>>>>>>>> + * immediately as if the virtual IRQ were completed
>>>>>>>> + * by the guest
>>>>>>>> + */
>>>>>>>> +kvm_notify_acked_irq(kvm, 0, irq);
>>>>>>>> +return -EAGAIN;
>>>>>>>
>>>>>>> This looks fishy, I don't fully understand which case you're catering
>>>>>>> towards here (the comment is hard to understand), but my gut feeling is
>>>>>>> that you should back out (probably with an error) if the vgic is not
>>>>>>> initialized when calling this function. Setting up the routing in the
>>>>>>> first place should probably error out if the vgic is not available.
>>>>>> The issue is related to integration with QEMU and VFIO.
>>>>>> Currently VFIO signaling (we tell VFIO to signal the eventfd on a
>>>>>> peculiar physical IRQ) and irqfd setup (we tell KVM/IRQFD to handle
>>>>>> previous eventfd when triggered and inject a GSI) are done by QEMU
>>>>>> *before* the first vcpu run. so before VGIC is ready.
>>>>>>
>>>>>> Both are done in a so called QEMU machine init done notifier. It could
>>>>>> be done in a QEMU reset notifier but I guess the problem would be the
>>>>>> same. and I think the same at which QEMU initializes it is correct.
>>>>>>
>>>>>> As soon as both VFIO signaling and irqfd are setup, virtual IRQ are
>>>>>> likely to be injected and this is what happens on some circumstances.
>>>>>> This happens on the 2d QEMU run after killing the 1st QEMU session. For
>>>>>> some reason I guess the HW device - in my case the xgmac - was released
>>>>>> in such a way the IRQ wire was not reset. So as soon as VFIO driver
>>>>>> calls request_irq, the IRQ hits.
>>>>>>
>>>>>> I tried to change that this xgmac driver behavior but I must acknowledge
>>>>>> that for the time beeing I failed. I will continue investigating that.
>>>&g
Re: [RFC PATCH 4/5] ARM: enable linking against eventfd and irqchip
On 11/24/2014 10:27 PM, Nikolay Nikolaev wrote: > This enables compilation of the eventfd feature on ARM. Hi Nikolay, why irqchip in the title? Best Regards Eric > > Signed-off-by: Nikolay Nikolaev > --- > arch/arm/kvm/Kconfig |1 + > arch/arm/kvm/Makefile |2 +- > 2 files changed, 2 insertions(+), 1 deletion(-) > > diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig > index 466bd29..a4b0312 100644 > --- a/arch/arm/kvm/Kconfig > +++ b/arch/arm/kvm/Kconfig > @@ -20,6 +20,7 @@ config KVM > bool "Kernel-based Virtual Machine (KVM) support" > select PREEMPT_NOTIFIERS > select ANON_INODES > + select HAVE_KVM_EVENTFD > select HAVE_KVM_CPU_RELAX_INTERCEPT > select KVM_MMIO > select KVM_ARM_HOST > diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile > index 443b8be..539c1a5 100644 > --- a/arch/arm/kvm/Makefile > +++ b/arch/arm/kvm/Makefile > @@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt) > AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt) > > KVM := ../../../virt/kvm > -kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o > +kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o > > obj-y += kvm-arm.o init.o interrupts.o > obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 1/5] KVM: redesing kvm_io_bus_ API to pass VCPU structure to the callbacks.
Hi Nikolay,
typo in the title,
On 11/24/2014 10:26 PM, Nikolay Nikolaev wrote:
> This is needed in e.g. ARM vGIC emulation, where the MMIO handling
> depends on the VCPU that does the access.
>
> Signed-off-by: Nikolay Nikolaev
> ---
> arch/ia64/kvm/kvm-ia64.c |4 ++--
> arch/powerpc/kvm/powerpc.c |4 ++--
> arch/s390/kvm/diag.c |2 +-
> arch/x86/kvm/vmx.c |2 +-
> arch/x86/kvm/x86.c | 11 ++-
> include/linux/kvm_host.h | 10 +-
> virt/kvm/coalesced_mmio.c |5 +++--
> virt/kvm/eventfd.c |4 ++--
> virt/kvm/iodev.h | 23 +++
> virt/kvm/kvm_main.c| 32
> 10 files changed, 53 insertions(+), 44 deletions(-)
>
> diff --git a/arch/ia64/kvm/kvm-ia64.c b/arch/ia64/kvm/kvm-ia64.c
> index ec6b9ac..16f1d26 100644
> --- a/arch/ia64/kvm/kvm-ia64.c
> +++ b/arch/ia64/kvm/kvm-ia64.c
> @@ -246,10 +246,10 @@ static int handle_mmio(struct kvm_vcpu *vcpu, struct
> kvm_run *kvm_run)
> return 0;
> mmio:
> if (p->dir)
> - r = kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, p->addr,
> + r = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, p->addr,
> p->size, &p->data);
> else
> - r = kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, p->addr,
> + r = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, p->addr,
>p->size, &p->data);
> if (r)
> printk(KERN_ERR"kvm: No iodevice found! addr:%lx\n", p->addr);
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index c1f8f53..5ac065b 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -814,7 +814,7 @@ int kvmppc_handle_load(struct kvm_run *run, struct
> kvm_vcpu *vcpu,
>
> idx = srcu_read_lock(&vcpu->kvm->srcu);
>
> - ret = kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, run->mmio.phys_addr,
> + ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, run->mmio.phys_addr,
> bytes, &run->mmio.data);
>
> srcu_read_unlock(&vcpu->kvm->srcu, idx);
> @@ -887,7 +887,7 @@ int kvmppc_handle_store(struct kvm_run *run, struct
> kvm_vcpu *vcpu,
>
> idx = srcu_read_lock(&vcpu->kvm->srcu);
>
> - ret = kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, run->mmio.phys_addr,
> + ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, run->mmio.phys_addr,
> bytes, &run->mmio.data);
>
> srcu_read_unlock(&vcpu->kvm->srcu, idx);
> diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/diag.c
> index 9254aff..329ec75 100644
> --- a/arch/s390/kvm/diag.c
> +++ b/arch/s390/kvm/diag.c
> @@ -213,7 +213,7 @@ static int __diag_virtio_hypercall(struct kvm_vcpu *vcpu)
>* - gpr 3 contains the virtqueue index (passed as datamatch)
>* - gpr 4 contains the index on the bus (optionally)
>*/
> - ret = kvm_io_bus_write_cookie(vcpu->kvm, KVM_VIRTIO_CCW_NOTIFY_BUS,
> + ret = kvm_io_bus_write_cookie(vcpu, KVM_VIRTIO_CCW_NOTIFY_BUS,
> vcpu->run->s.regs.gprs[2] & 0x,
> 8, &vcpu->run->s.regs.gprs[3],
> vcpu->run->s.regs.gprs[4]);
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index 3e556c6..e6d9f01 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5620,7 +5620,7 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
> gpa_t gpa;
>
> gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
> - if (!kvm_io_bus_write(vcpu->kvm, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
> + if (!kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
> skip_emulated_instruction(vcpu);
> return 1;
> }
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 0033df3..bbf9375 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -4053,7 +4053,7 @@ static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t
> addr, int len,
> n = min(len, 8);
> if (!(vcpu->arch.apic &&
> !kvm_iodevice_write(&vcpu->arch.apic->dev, addr, n, v))
vcpu param should be added above too.
Eric
> - && kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, addr, n, v))
> + && kvm_io_bus_write(vcpu, KVM_MMIO_BUS, addr, n, v))
> break;
> handled += n;
> addr += n;
> @@ -4072,8 +4072,9 @@ static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t
> addr, int len, void *v)
> do {
> n = min(len, 8);
> if (!(vcpu->arch.apic &&
> - !kvm_iodevice_read(&vcpu->arch.apic->dev, addr, n, v))
> - && kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, addr, n, v))
> + !kvm_iodevice_read(vcpu, &vcpu->arch.apic->dev,
> + addr, n, v))
> + && kvm_io_bus_r
Re: [RFC PATCH 2/5] ARM: on IO mem abort - route the call to KVM MMIO bus
On 11/24/2014 10:26 PM, Nikolay Nikolaev wrote:
> On IO memory abort, try to handle the MMIO access thorugh the KVM
typo
> registered read/write callbacks. This is done by invoking the relevant
> kvm_io_bus_* API.
>
> Signed-off-by: Nikolay Nikolaev
> ---
> arch/arm/kvm/mmio.c | 33 +
> 1 file changed, 33 insertions(+)
>
> diff --git a/arch/arm/kvm/mmio.c b/arch/arm/kvm/mmio.c
> index 4cb5a93..81230da 100644
> --- a/arch/arm/kvm/mmio.c
> +++ b/arch/arm/kvm/mmio.c
> @@ -162,6 +162,36 @@ static int decode_hsr(struct kvm_vcpu *vcpu, phys_addr_t
> fault_ipa,
> return 0;
> }
>
> +/**
> + * kvm_handle_mmio - handle an in-kernel MMIO access
handle_kernel_mmio
> + * @vcpu:pointer to the vcpu performing the access
> + * @run: pointer to the kvm_run structure
> + * @mmio:pointer to the data describing the access
> + *
> + * returns true if the MMIO access has been performed in kernel space,
> + * and false if it needs to be emulated in user space.
> + */
> +static bool handle_kernel_mmio(struct kvm_vcpu *vcpu, struct kvm_run *run,
> + struct kvm_exit_mmio *mmio)
> +{
> + int ret;
> +
> + if (mmio->is_write) {
> + ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, mmio->phys_addr,
> + mmio->len, &mmio->data);
> +
> + } else {
> + ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, mmio->phys_addr,
> + mmio->len, &mmio->data);
> + }
> + if (!ret) {
> + kvm_prepare_mmio(run, mmio);
> + kvm_handle_mmio_return(vcpu, run);
Shouldn't the comment associated to kvm_handle_mmio_return saying
the function "should only be called after returning from userspace for
MMIO load emulation" be updated? Here we did kernel emulation and we
call it. also in vgic it is called. Must aknowledge I do not fully
understand what the .rt is.
> + }
> +
> + return !ret;
what if kvm_io_bus_read returned -EOPNOTSUPP?
> +}
> +
> int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run,
>phys_addr_t fault_ipa)
> {
> @@ -200,6 +230,9 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run
> *run,
> if (vgic_handle_mmio(vcpu, run, &mmio))
> return 1;
>
> + if (handle_kernel_mmio(vcpu, run, &mmio))
> + return 1;
> +
> kvm_prepare_mmio(run, &mmio);
> return 0;
> }
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 3/5] KVM: ARM VGIC add kvm_io_bus_ frontend
On 11/24/2014 10:26 PM, Nikolay Nikolaev wrote:
> In io_mem_abort remove the call to vgic_handle_mmio. The target is to have
> a single MMIO handling path - that is through the kvm_io_bus_ API.
>
> Register a kvm_io_device in kvm_vgic_init on the whole vGIC MMIO region.
> Both read and write calls are redirected to vgic_io_dev_access where
> kvm_exit_mmio is composed to pass it to vm_ops.handle_mmio.
>
>
> Signed-off-by: Nikolay Nikolaev
> ---
> arch/arm/kvm/mmio.c|3 --
> include/kvm/arm_vgic.h |3 +-
> virt/kvm/arm/vgic.c| 88
>
> 3 files changed, 74 insertions(+), 20 deletions(-)
>
> diff --git a/arch/arm/kvm/mmio.c b/arch/arm/kvm/mmio.c
> index 81230da..1c44a2b 100644
> --- a/arch/arm/kvm/mmio.c
> +++ b/arch/arm/kvm/mmio.c
> @@ -227,9 +227,6 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run
> *run,
> if (mmio.is_write)
> mmio_write_buf(mmio.data, mmio.len, data);
>
> - if (vgic_handle_mmio(vcpu, run, &mmio))
> - return 1;
> -
> if (handle_kernel_mmio(vcpu, run, &mmio))
> return 1;
>
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index e452ef7..d9b7d2a 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -233,6 +233,7 @@ struct vgic_dist {
> unsigned long *irq_pending_on_cpu;
>
> struct vgic_vm_ops vm_ops;
> + struct kvm_io_device*io_dev;
> #endif
> };
>
> @@ -307,8 +308,6 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid,
> unsigned int irq_num,
> bool level);
> void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg);
> int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu);
> -bool vgic_handle_mmio(struct kvm_vcpu *vcpu, struct kvm_run *run,
> - struct kvm_exit_mmio *mmio);
>
> #define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel))
> #define vgic_initialized(k) ((k)->arch.vgic.ready)
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 1213da5..3da1115 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -31,6 +31,9 @@
> #include
> #include
> #include
> +#include
> +
> +#include "iodev.h"
>
> /*
> * How the whole thing works (courtesy of Christoffer Dall):
> @@ -775,28 +778,81 @@ bool vgic_handle_mmio_range(struct kvm_vcpu *vcpu,
> struct kvm_run *run,
> return true;
> }
>
> -/**
> - * vgic_handle_mmio - handle an in-kernel MMIO access for the GIC emulation
> - * @vcpu: pointer to the vcpu performing the access
> - * @run: pointer to the kvm_run structure
> - * @mmio: pointer to the data describing the access
> - *
> - * returns true if the MMIO access has been performed in kernel space,
> - * and false if it needs to be emulated in user space.
> - * Calls the actual handling routine for the selected VGIC model.
> - */
> -bool vgic_handle_mmio(struct kvm_vcpu *vcpu, struct kvm_run *run,
> - struct kvm_exit_mmio *mmio)
may be worth to restore some comments
> +static int vgic_io_dev_access(struct kvm_vcpu *vcpu, struct kvm_io_device
> *this,
> + gpa_t addr, int len, void *val, bool is_write)
> {
> - if (!irqchip_in_kernel(vcpu->kvm))
> - return false;
> + struct kvm_exit_mmio mmio;
> + bool ret;
> +
> + mmio = (struct kvm_exit_mmio) {
> + .phys_addr = addr,
> + .len = len,
> + .is_write = is_write,
> + };
> +
> + if (is_write)
> + memcpy(mmio.data, val, len);
>
> /*
>* This will currently call either vgic_v2_handle_mmio() or
>* vgic_v3_handle_mmio(), which in turn will call
>* vgic_handle_mmio_range() defined above.
>*/
> - return vcpu->kvm->arch.vgic.vm_ops.handle_mmio(vcpu, run, mmio);
> + ret = vcpu->kvm->arch.vgic.vm_ops.handle_mmio(vcpu, vcpu->run, &mmio);
> +
> + if (!is_write)
> + memcpy(val, mmio.data, len);
> +
> + return ret ? 0 : 1;
> +}
> +
> +static int vgic_io_dev_read(struct kvm_vcpu *vcpu, struct kvm_io_device
> *this,
> + gpa_t addr, int len, void *val)
> +{
> + return vgic_io_dev_access(vcpu, this, addr, len, val, false);
> +}
> +
> +static int vgic_io_dev_write(struct kvm_vcpu *vcpu, struct kvm_io_device
> *this,
> +gpa_t addr, int len, const void *val)
> +{
> + return vgic_io_dev_access(vcpu, this, addr, len, (void *)val, true);
> +}
> +
> +static const struct kvm_io_device_ops vgic_io_dev_ops = {
> + .read = vgic_io_dev_read,
> + .write = vgic_io_dev_write,
> +};
> +
> +static int vgic_register_kvm_io_dev(struct kvm *kvm)
> +{
> + struct kvm_io_device *dev;
> + int ret;
> +
> + struct vgic_dist *dist = &kvm->arch.vgic;
> + unsigned long base = dist->vgic_dist_base;
Nikolay,
are you sure dist->vgic_dist_base was set by the guest?
in vgic_init/vm_ops.vgic_init (vg
Re: [RFC PATCH 0/5] ARM: KVM: Enable the ioeventfd capability of KVM on ARM
On 11/24/2014 10:26 PM, Nikolay Nikolaev wrote: > The IOEVENTFD KVM capability is a prerequisite for vhost support, > and is also used to implement improved interrupt handling in VFIO drivers. Hi Nikolay As far as I am aware, irqfd currently is used in vfio context, not ioeventfd, although they both are implemented in eventfd.c. Could you elaborate? > > This series enables the ioeventfd KVM capability on ARM. > > The implementation routes MMIO access in the IO abort handler to the KVM IO > bus. > If there is already a registered ioeventfd handler for this address, the file > descriptor will be triggered. > > We extended the KVM IO bus API to expose the VCPU struct pointer. Now the VGIC > MMIO access is done through this API. For this to operate the VGIC registers a > kvm_io_device which reprresents the whole MMIO region. typo & the whole dist MMIO region? I think the transformation makes sense and does not bring upheavals in the vgic code which is good. Best Regards Eric > > The patches are implemented on top of the latest Andre's vGICv3 work from > here: > http://www.linux-arm.org/git?p=linux-ap.git;a=shortlog;h=refs/heads/kvm-gicv3/v4 > > The code was tested on Dual Cortex-A15 Exynos5250 (ARM Chromebook). > > --- > > Nikolay Nikolaev (5): > KVM: redesing kvm_io_bus_ API to pass VCPU structure to the callbacks. > ARM: on IO mem abort - route the call to KVM MMIO bus > KVM: ARM VGIC add kvm_io_bus_ frontend > ARM: enable linking against eventfd and irqchip > ARM: enable KVM_CAP_IOEVENTFD > > > arch/arm/kvm/Kconfig |1 + > arch/arm/kvm/Makefile |2 + > arch/arm/kvm/arm.c |3 ++ > arch/arm/kvm/mmio.c| 32 > arch/ia64/kvm/kvm-ia64.c |4 +- > arch/powerpc/kvm/powerpc.c |4 +- > arch/s390/kvm/diag.c |2 + > arch/x86/kvm/vmx.c |2 + > arch/x86/kvm/x86.c | 11 +++--- > include/kvm/arm_vgic.h |3 +- > include/linux/kvm_host.h | 10 +++-- > virt/kvm/arm/vgic.c| 88 > +--- > virt/kvm/coalesced_mmio.c |5 ++- > virt/kvm/eventfd.c |4 +- > virt/kvm/iodev.h | 23 > virt/kvm/kvm_main.c| 32 > 16 files changed, 163 insertions(+), 63 deletions(-) > > -- > Signature > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH v2 1/2] KVM: kvm-vfio: User API for VT-d Posted-Interrupts
On 11/25/2014 05:10 PM, Alex Williamson wrote:
> On Tue, 2014-11-25 at 16:01 +0100, Eric Auger wrote:
>> On 11/25/2014 01:23 PM, Feng Wu wrote:
>>> This patch adds and documents a new attribute
>>> KVM_DEV_VFIO_DEVICE_POSTING_IRQ in KVM_DEV_VFIO_DEVICE group.
>>> This new attribute is used for VT-d Posted-Interrupts.
>>>
>>> When guest OS changes the interrupt configuration for an
>>> assigned device, such as, MSI/MSIx data/address fields,
>>> QEMU will use this IRQ attribute to tell KVM to update the
>>> related IRTE according the VT-d Posted-Interrrupts Specification,
>>> such as, the guest vector should be updated in the related IRTE.
>>>
>>> Signed-off-by: Feng Wu
>>> ---
>>> Documentation/virtual/kvm/devices/vfio.txt |9 +
>>> include/uapi/linux/kvm.h | 10 ++
>>> 2 files changed, 19 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
>>> b/Documentation/virtual/kvm/devices/vfio.txt
>>> index f7aff29..39dee86 100644
>>> --- a/Documentation/virtual/kvm/devices/vfio.txt
>>> +++ b/Documentation/virtual/kvm/devices/vfio.txt
>>> @@ -42,3 +42,12 @@ activated before VFIO_DEVICE_SET_IRQS has been called to
>>> trigger the IRQ
>>> or associate an eventfd to it. Unforwarding can only be called while the
>>> signaling has been disabled with VFIO_DEVICE_SET_IRQS. If this condition is
>>> not satisfied, the command returns an -EBUSY.
>>> +
>>> + KVM_DEV_VFIO_DEVICE_POSTING_IRQ: Use posted interrtups mechanism to post
>>> + the IRQ to guests.
>>> +For this attribute, kvm_device_attr.addr points to a kvm_posted_intr
>>> struct.
>>> +
>>> +When guest OS changes the interrupt configuration for an assigned device,
>>> +such as, MSI/MSIx data/address fields, QEMU will use this IRQ attribute
>>> +to tell KVM to update the related IRTE according the VT-d
>>> Posted-Interrrupts
>>> +Specification, such as, the guest vector should be updated in the related
>>> IRTE.
>>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>>> index a269a42..e5f86ad 100644
>>> --- a/include/uapi/linux/kvm.h
>>> +++ b/include/uapi/linux/kvm.h
>>> @@ -949,6 +949,7 @@ struct kvm_device_attr {
>>> #define KVM_DEV_VFIO_DEVICE 2
>>> #define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
>>> #define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
>>> +#define KVM_DEV_VFIO_DEVICE_POSTING_IRQ 3
>>>
>>> enum kvm_device_type {
>>> KVM_DEV_TYPE_FSL_MPIC_20= 1,
>>> @@ -973,6 +974,15 @@ struct kvm_arch_forwarded_irq {
>>> __u32 gsi; /* gsi, ie. virtual IRQ number */
>>> };
>>>
Hi Feng, Alex,
I am currently reworking my code to use something closer to this struct.
Would you agree with following changes?
>>> +struct kvm_posted_intr {
kvm_posted_irq
>>> + __u32 argsz;
>>> + __u32 fd; /* file descriptor of the VFIO device */
>>> + __u32 index; /* VFIO device IRQ index */
>>> + __u32 start;
>>> + __u32 count;
>>> + int virq[0];/* gsi, ie. virtual IRQ number */
__u32 gsi[];
>>> +};
>> Hi Feng,
>>
>> This struct could be used by arm code too. If Alex agrees I could use
>> that one instead. We just need to find a common sensible name
>
> Yep, the interface might as well support batch setup. The vfio code
> uses -1 for teardown if we want to avoid FORWARD vs UNFORWARD we could
> let the data in the structure define which operation to do.
In case we remove the unforward and use fd=1 to tear down, the virq=gsi
must uniquely identify the struct. For ARM I think this is true, we
cannot have several physical IRQ forwarded to the same GSI. I don't know
about posted irqs or other archs.
Best Regards
Eric
Ideally the
> code in virt/kvm/vfio.c would be almost entirely shared and just make
> different arch_foo() callouts. The PCI smarts in 2/2 here should
> probably be moved out to that same arch_ code. Thanks,
>
> Alex
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 3/8] VFIO: platform: forwarded state tested when selecting IRQ handler
Hi Christoffer,
On 11/30/2014 01:47 PM, Christoffer Dall wrote:
> The subject reads strangely, perhaps just:
>
> VFIO: platform: test forward state when selecting IRQ handler
OK
>
> On Sun, Nov 23, 2014 at 07:35:55PM +0100, Eric Auger wrote:
>> In case the IRQ is forwarded, the VFIO platform IRQ handler does not
>> need to disable the IRQ anymore.
>>
>> When setting the IRQ handler we now also test the forwarded state. In
>> case the IRQ is forwarded we select the edge handler (no automaske).
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v2 -> v3:
>> - forwarded state was tested in the handler. Now the forwarded state
>> is tested before setting the handler. This definitively limits
>> the dynamics of forwarded state changes but I don't think there is
>> a use case where we need to be able to change the state at any time.
>
> user space can change this by calling the VFIO_IRQ_SET_ACTION_TRIGGER
> whenever it wants, right?
yes the user can set/unset the VFIO signaling (and request_irq/free_irq)
using VFIO_IRQ_SET_ACTION_TRIGGER. In this new version I do not allow
changing the forwarded state when the handler is attached (request_irq).
Does it answer your interrogation?
Best Regards
Eric
>
>> ---
>> drivers/vfio/platform/vfio_platform_irq.c | 7 ++-
>> 1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/vfio/platform/vfio_platform_irq.c
>> b/drivers/vfio/platform/vfio_platform_irq.c
>> index 08d400e..61a2920 100644
>> --- a/drivers/vfio/platform/vfio_platform_irq.c
>> +++ b/drivers/vfio/platform/vfio_platform_irq.c
>> @@ -230,8 +230,13 @@ static int vfio_platform_set_irq_trigger(struct
>> vfio_platform_device *vdev,
>> {
>> struct vfio_platform_irq *irq = &vdev->irqs[index];
>> irq_handler_t handler;
>> +struct irq_data *d;
>> +bool is_forwarded;
>>
>> -if (vdev->irqs[index].flags & VFIO_IRQ_INFO_MASKABLE)
>> +d = irq_get_irq_data(irq->hwirq);
>> +is_forwarded = irqd_irq_forwarded(d);
>> +
>> +if ((vdev->irqs[index].flags & VFIO_IRQ_INFO_MASKABLE) && !is_forwarded)
>> handler = vfio_maskable_irq_handler;
>> else
>> handler = vfio_irq_handler;
>> --
>> 1.9.1
>>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 4/8] KVM: kvm-vfio: User API for IRQ forwarding
On 11/30/2014 01:53 PM, Christoffer Dall wrote:
> On Sun, Nov 23, 2014 at 07:35:56PM +0100, Eric Auger wrote:
>> This patch adds and document a new KVM_DEV_VFIO_DEVICE group
>
> documents
OK
>
>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
>> to set a VFIO device IRQ as forwarded or not forwarded.
>> the command takes as argument a handle to a new struct named
>> kvm_arch_forwarded_irq.
>>
>> Signed-off-by: Eric Auger
>>
>> ---
>>
>> v2 -> v3:
>> - rework vfio kvm device documentation
>> - reword commit message and title
>> - add subindex in kvm_arch_forwarded_irq to be closer to VFIO API
>> - forwarding state can only be changed with VFIO IRQ signaling is off
>>
>> v1 -> v2:
>> - struct kvm_arch_forwarded_irq moved from arch/arm/include/uapi/asm/kvm.h
>> to include/uapi/linux/kvm.h
>> also irq_index renamed into index and guest_irq renamed into gsi
>> - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD
>> ---
>> Documentation/virtual/kvm/devices/vfio.txt | 34
>> --
>> include/uapi/linux/kvm.h | 10 +
>> 2 files changed, 38 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
>> b/Documentation/virtual/kvm/devices/vfio.txt
>> index ef51740..f7aff29 100644
>> --- a/Documentation/virtual/kvm/devices/vfio.txt
>> +++ b/Documentation/virtual/kvm/devices/vfio.txt
>> @@ -4,15 +4,24 @@ VFIO virtual device
>> Device types supported:
>>KVM_DEV_TYPE_VFIO
>>
>> -Only one VFIO instance may be created per VM. The created device
>> -tracks VFIO groups in use by the VM and features of those groups
>> -important to the correctness and acceleration of the VM. As groups
>> -are enabled and disabled for use by the VM, KVM should be updated
>> -about their presence. When registered with KVM, a reference to the
>> -VFIO-group is held by KVM.
>> +Only one VFIO instance may be created per VM.
>> +
>> +The created device tracks VFIO groups in use by the VM and features
>> +of those groups important to the correctness and acceleration of
>> +the VM. As groups are enabled and disabled for use by the VM, KVM
>> +should be updated about their presence. When registered with KVM,
>> +a reference to the VFIO-group is held by KVM.
>> +
>> +The device also tracks & enable VFIO device forwarded IRQs, if any.
>
> s/tracks & enable/tracks and enables/
ok
>
>> +A physical forwarded IRQ is directly completed by the guest. This
>> +requires HW support in the interrupt controller which must be able
>> +to automatically complete the physical IRQ when it detects the guest
>> +has completed the corresponding virtual IRQ. The modality sometimes
>> +is named direct EOI.
>
> this last sentence is a bit floating out there; perhaps you want to be
> more specific and say what it is on ARM.
Well this is naming some people used at the KVM forum. I thought this
was something known on other archs but I can drop this comment.
>
>>
>> Groups:
>>KVM_DEV_VFIO_GROUP
>> + KVM_DEV_VFIO_DEVICE
>>
>> KVM_DEV_VFIO_GROUP attributes:
>>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
>> @@ -20,3 +29,16 @@ KVM_DEV_VFIO_GROUP attributes:
>>
>> For each, kvm_device_attr.addr points to an int32_t file descriptor
>> for the VFIO group.
>> +
>> +KVM_DEV_VFIO_DEVICE attributes:
>> + KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as forwarded
>> + KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as not forwarded
>> +
>> +For each, kvm_device_attr.addr points to a kvm_arch_forwarded_irq struct.
>> +
>> +The forwarded state can only be changed when the VFIO signaling mechanism
>> +for this physical IRQ is not set. In other words, forwarding must be
>> +activated before VFIO_DEVICE_SET_IRQS has been called to trigger the IRQ
>> +or associate an eventfd to it. Unforwarding can only be called while the
>> +signaling has been disabled with VFIO_DEVICE_SET_IRQS. If this condition is
>> +not satisfied, the command returns an -EBUSY.
>
> s/command/ioctl/
ok
>
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 6076882..a269a42 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -946,6 +946,9 @@ struct kvm_device_attr {
>> #define KVM_DEV_VFIO_GROUP 1
>> #define KVM_DEV_VFIO_GROUP_ADD
Re: [PATCH v3 2/8] KVM: arm64: Enable the KVM-VFIO device
On 11/30/2014 01:14 PM, Christoffer Dall wrote: > On Sun, Nov 23, 2014 at 07:35:54PM +0100, Eric Auger wrote: >> Used by KVM-enabled VFIO-based device passthrough support in QEMU. >> >> Signed-off-by: Joel Schopp >> Signed-off-by: Eric Auger >> >> --- >> >> Extracted from [RFC PATCH] arm64: KVM: add irqfd support >> http://www.spinics.net/lists/kvm-arm/msg10798.html >> --- >> arch/arm64/kvm/Kconfig | 1 + >> arch/arm64/kvm/Makefile | 2 +- >> 2 files changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig >> index 09c25c2..2edf926 100644 >> --- a/arch/arm64/kvm/Kconfig >> +++ b/arch/arm64/kvm/Kconfig >> @@ -26,6 +26,7 @@ config KVM >> select KVM_ARM_HOST >> select KVM_ARM_VGIC >> select KVM_ARM_TIMER >> +select KVM_VFIO >> select HAVE_KVM_EVENTFD >> ---help--- >>Support hosting virtualized guest machines. >> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile >> index 2e6b827..81ed091 100644 >> --- a/arch/arm64/kvm/Makefile >> +++ b/arch/arm64/kvm/Makefile >> @@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm >> >> obj-$(CONFIG_KVM_ARM_HOST) += kvm.o >> >> -kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o >> $(KVM)/eventfd.o >> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o >> $(KVM)/eventfd.o $(KVM)/vfio.o >> kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o >> kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o >> >> -- >> 1.9.1 >> > > Should these patches not be squashed into one? Yes I can do. > > Also, what do they enable at this point? Should they be queued by the > end of the series instead? Well to me this patch should be moved even outside of this series. The KVM-VFIO device is loaded when the QEMU VFIO device is instantiated. This is used to record the VFIO groups in use. In VFIO platform case, if the KVM-VFIO device does not exist, this is not fatal but we get a warning in QEMU. The KVM-VFIO device however is mandatory to enable forwarded irq feature. Best Regards Eric > > -Christoffer > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
