Tested-by: Lei Yao <lei.a....@intel.com>
Apply patch to dpdk_next_virtio branch.
Qemu version: 2.5.0
Kernel version in VM: 4.8.1
Following TCs are tested and passed:
Test Case1: Basic Virtio Interrupt test
Test Case2: Interrupted received in VM with different Virtio version(0.95 and
1.0)
Test Case3: Interrupted by packet data on all queue
Test Case4: Interrupted by packet data on unique queue
Test Case5: Stop packet transmit, related cores will be back to sleep
> -----Original Message-----
> From: Tan, Jianfeng
> Sent: Monday, January 16, 2017 10:47 PM
> To: dev@dpdk.org
> Cc: yuanhan....@linux.intel.com; step...@networkplumber.org; Yao, Lei A
> <lei.a....@intel.com>; Tan, Jianfeng <jianfeng....@intel.com>
> Subject: [PATCH v3 00/10] rxq interrupt mode for virtio PMD
>
> v3:
> - Update documents:
> * doc/guides/nics/features/virtio.ini
> * doc/guides/nics/features/virtio_vec.ini
> * doc/guides/nics/virtio.rst
> - Use hw->max_queue_pairs instead of dev->data->nb_rx_queues to
> allocate intr_vec array.
> - Fix v2 not working on legacy virtio devices by moving msix enabling
> before queue/irq binding.
> - Reword cover letter to give an overview of this series.
> - Remove wrapper to call vtpci->set_config_irq and vtpci->set_queue_irq.
> - Rebase on the new code, and fix a bug after changes by the commit
> bb30369dc10("eal: allow passing const interrupt handle"). Basically,
> it changes the way to get max interrupts. And we need to re-register
> callback to update intr_handle->max_intr.
> - In l3fwd-power ptype fix, use rte_eth_dev_get_supported_ptypes() to
> query if PMD provides needed ptypes.
>
> v2:
> - Add PCI queue/irq config ops.
> - Move rxq interrupt settings before sending DRIVER OK.
>
> Historically, virtio PMD can only be binded to igb_uio or
> uio_pci_generic, and not for vfio-pci (iommu group cannot be created as
> vIOMMU is not enabled in QEMU yet). Besides, quote from
> http://dpdk.org/doc/guides-16.11/rel_notes/release_2_1.html:
> "Per queue RX interrupt events are only allowed in VFIO
> which supports multiple MSI-X vectors."
>
> Linux starts to support VFIO NO-IOMMU mode since 4.8.0. It cannot put
> devices into groups for separation as normal VFIO does. So it does not
> require QEMU to support vIOMMU. But it does inherit other benefits from
> VFIO framework, like better interrupts supports (than UIO). It gives a
> good chance to enable rxq interrupt for virtio PMD.
>
> To implement it,
> a. Firstly, we need to enable msix. This should be done before DRIVER_OK
> setting and also before queue/irq binding in step b.
> b. Bind queue/irq through portio (legacy devices) or mmio (modern devices).
> So far, we hard-code 1:1 queue/irq mapping (each rx queue has one
> exclusive interrupt), like this:
> vec 0 -> config irq
> vec 1 -> rxq0
> vec 2 -> rxq1
> ...
>
> which means, the "vectors" option of QEMU should be configured with
> a value >= N+1 (N is the number of the queue pairs).
> c. To enable/disable interrupt notification, flags on virtqueues are used
> to control devices either sending interrupt or not.
> d. Encap above behaviors into callbacks in ether_dev_ops, like
> rx_queue_intr_enable/rx_queue_intr_disable/rx_descriptor_done etc.
>
>
> How to test:
>
> Step 1, prepare a VM image with kernel version >= 4.8.0, and make sure
> the kernel is compiled with CONFIG_VFIO_NOIOMMU=y.
>
> Step 2, on the host, start a testpmd with a vhost port:
> $ testpmd -c 0x7 -m 1024 --vdev 'eth_vhost0,iface=/tmp/sock0,queues=2' \
> --no-pci -- -i --rxq=2 --txq=2 --nb-cores=2
>
> Step 3, boot the VM:
> $ qemu ... -chardev socket,id=chr1,path=/tmp/sock0 \
> -netdev vhost-user,id=net1,chardev=chr1,vhostforce,queues=2 \
> -device virtio-net-pci,netdev=net1,mq=on,vectors=5 ...
>
> Step 4, insert kernel modules
> $ modprobe vfio enable_unsafe_noiommu_mode=1
> $ modprobe vfio-pci
>
> Step 5, start l3fwd-power in VM:
> $ l3fwd-power -c 0x3 -n 4 -- -p 1 -P --config="(0,0,1),(0,1,1)" \
> --no-numa --parse-ptype
>
> Step 6, send packets from testpmd on the host:
> $ start tx_first
>
> Then l3fwd-power outputs:
> L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 0
> L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 1
>
> Signed-off-by: Jianfeng Tan <jianfeng....@intel.com>
>
> Jianfeng Tan (10):
> net/virtio: fix rewriting LSC flag
> net/virtio: clean up wrapper of set_config_irq
> net/virtio: add Rx descriptor check
> net/virtio: add PCI ops for queue/irq binding
> net/virtio: add Rx queue intr enable/disable functions
> net/virtio: setup rxq interrupts
> net/virtio: unbind intr/eventfd when stop device
> net/virtio: unmapping queue/irq when close device
> examples/l3fwd-power: add parse-ptype option
> examples/l3fwd-power: fix not stop and close device
>
> drivers/net/virtio/virtio_ethdev.c | 161
> +++++++++++++++++++++++++++++++++++--
> drivers/net/virtio/virtio_ethdev.h | 3 +
> drivers/net/virtio/virtio_pci.c | 31 +++++--
> drivers/net/virtio/virtio_pci.h | 5 +-
> drivers/net/virtio/virtio_rxtx.c | 9 +++
> drivers/net/virtio/virtqueue.c | 11 ---
> drivers/net/virtio/virtqueue.h | 16 +++-
> examples/l3fwd-power/main.c | 120 ++++++++++++++++++++++++++-
> 8 files changed, 325 insertions(+), 31 deletions(-)
>
> --
> 2.7.4
