> -----Original Message-----
> From: Nithin Dabilpuram <ndabilpu...@marvell.com>
> Sent: Wednesday, July 17, 2019 1:44 AM
> To: Hyong Youb Kim (hyonkim) <hyon...@cisco.com>; David Marchand
> <david.march...@redhat.com>; Thomas Monjalon
> <tho...@monjalon.net>; Ferruh Yigit <ferruh.yi...@intel.com>; Bruce
> Richardson <bruce.richard...@intel.com>
> Cc: jer...@marvell.com; John Daley (johndale) <johnd...@cisco.com>;
> Shahed Shaikh <shsha...@marvell.com>; dev@dpdk.org; Nithin Dabilpuram
> <ndabilpu...@marvell.com>
> Subject: [RFC PATCH v3 2/3] eal: add mask and unmask interrupt APIs
>
> Add new mask and unmask interrupt APIs to avoid using
> VFIO_IRQ_SET_ACTION_TRIGGER for masking/unmasking purpose for VFIO
> based handlers. This implementation is specific to Linux.
>
> Using action trigger for masking and unmasking has below issues
>
> * Time consuming to do for every interrupt received as it will
> free_irq() followed by request_irq() and all other initializations
> * A race condition because of a window between free_irq() and
> request_irq() with packet reception still on and device still
> enabled and would throw warning messages like below.
> [158764.159833] do_IRQ: 9.34 No irq handler for vector
>
> In this patch, mask/unmask is a no-op for VFIO_MSIX/VFIO_MSI interrupts
> as they are edge triggered and kernel would not mask the interrupt before
> delivering the event to userspace.
>
> Signed-off-by: Nithin Dabilpuram <ndabilpu...@marvell.com>
> ---
> v3:
> * Re-org patch to move driver change to 3/3
> * Add stub implementation for freebsd.
> * Fix version map file for new apis.
> v2:
> Make change in qede driver for unmask
> lib/librte_eal/common/include/rte_interrupts.h | 23 ++++
> lib/librte_eal/freebsd/eal/eal_interrupts.c | 54 +++++++++
> lib/librte_eal/linux/eal/eal_interrupts.c | 155
> +++++++++++++++++++++++++
> lib/librte_eal/rte_eal_version.map | 2 +
> 4 files changed, 234 insertions(+)
>
> diff --git a/lib/librte_eal/common/include/rte_interrupts.h
> b/lib/librte_eal/common/include/rte_interrupts.h
> index c1e912c..b0675be 100644
> --- a/lib/librte_eal/common/include/rte_interrupts.h
> +++ b/lib/librte_eal/common/include/rte_interrupts.h
> @@ -118,6 +118,29 @@ int rte_intr_enable(const struct rte_intr_handle
> *intr_handle);
> */
> int rte_intr_disable(const struct rte_intr_handle *intr_handle);
>
> +/**
> + * It masks the interrupt for the specified handle.
> + *
> + * @param intr_handle
> + * pointer to the interrupt handle.
> + *
> + * @return
> + * - On success, zero.
> + * - On failure, a negative value.
> + */
> +int rte_intr_mask(const struct rte_intr_handle *intr_handle);
> +
> +/**
> + * It unmasks the interrupt for the specified handle.
> + *
> + * @param intr_handle
> + * pointer to the interrupt handle.
> + *
> + * @return
> + * - On success, zero.
> + * - On failure, a negative value.
> + */
> +int rte_intr_unmask(const struct rte_intr_handle *intr_handle);
> #ifdef __cplusplus
> }
> #endif
[...]
> diff --git a/lib/librte_eal/linux/eal/eal_interrupts.c
> b/lib/librte_eal/linux/eal/eal_interrupts.c
> index 79ad5e8..f619981 100644
> --- a/lib/librte_eal/linux/eal/eal_interrupts.c
> +++ b/lib/librte_eal/linux/eal/eal_interrupts.c
[...]
> int
> +rte_intr_mask(const struct rte_intr_handle *intr_handle)
> +{
> + if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV)
> + return 0;
> +
> + if (!intr_handle || intr_handle->fd < 0 || intr_handle->uio_cfg_fd < 0)
> + return -1;
> +
> + switch (intr_handle->type){
> + /* Both masking and disabling are same for UIO */
> + case RTE_INTR_HANDLE_UIO:
> + if (uio_intr_disable(intr_handle))
> + return -1;
> + break;
> + case RTE_INTR_HANDLE_UIO_INTX:
> + if (uio_intx_intr_disable(intr_handle))
> + return -1;
> + break;
> + /* not used at this moment */
> + case RTE_INTR_HANDLE_ALARM:
> + return -1;
> +#ifdef VFIO_PRESENT
> + case RTE_INTR_HANDLE_VFIO_MSIX:
> + case RTE_INTR_HANDLE_VFIO_MSI:
> + return 0;
Isn't this a little confusing? It returns success, but irq is not
masked.
As is, return code 0 means...
- igb_uio: irq is masked for INTx, MSI, MSI-X
- vfio-pci + INTx: irq is masked
- vfio-pci + MSI/MSI-X: no changes
Masking is useful only for INTx, IMO...
Masking MSI/MSI-X via PCI-defined mechanisms (e.g. Mask bit in MSI-X
Table) has no practical use for drivers. Handshaking/masking/unmasking
is done via device/vendor specific ways, as needed. See all those
ack/block/unblock/credit/... mechanisms used in various drivers/NICs
to control interrupts their own way.
A long time ago in early PCIe days, the linux kernel did auto-masking
for MSI/MSI-X (i.e. mask before calling netdev irq handler). It was
soon removed as it was unnecessary overhead (expensive PIOs to NIC for
every interrupt). Windows and FreeBSD do not do auto-masking either.
Thanks.
-Hyong
> + case RTE_INTR_HANDLE_VFIO_LEGACY:
> + if (vfio_mask_intx(intr_handle))
> + return -1;
> + break;
> +#ifdef HAVE_VFIO_DEV_REQ_INTERFACE
> + case RTE_INTR_HANDLE_VFIO_REQ:
> + return -1;
> +#endif
> +#endif
> + /* not used at this moment */
> + case RTE_INTR_HANDLE_DEV_EVENT:
> + return -1;
> + /* unknown handle type */
> + default:
> + RTE_LOG(ERR, EAL,
> + "Unknown handle type of fd %d\n",
> + intr_handle->fd);
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +int
> +rte_intr_unmask(const struct rte_intr_handle *intr_handle)
> +{
> + if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV)
> + return 0;
> +
> + if (!intr_handle || intr_handle->fd < 0 || intr_handle->uio_cfg_fd < 0)
> + return -1;
> +
> + switch (intr_handle->type){
> + /* Both unmasking and disabling are same for UIO */
> + case RTE_INTR_HANDLE_UIO:
> + if (uio_intr_enable(intr_handle))
> + return -1;
> + break;
> + case RTE_INTR_HANDLE_UIO_INTX:
> + if (uio_intx_intr_enable(intr_handle))
> + return -1;
> + break;
> + /* not used at this moment */
> + case RTE_INTR_HANDLE_ALARM:
> + return -1;
> +#ifdef VFIO_PRESENT
> + case RTE_INTR_HANDLE_VFIO_MSIX:
> + case RTE_INTR_HANDLE_VFIO_MSI:
> + return 0;
> + case RTE_INTR_HANDLE_VFIO_LEGACY:
> + if (vfio_unmask_intx(intr_handle))
> + return -1;
> + break;
> +#ifdef HAVE_VFIO_DEV_REQ_INTERFACE
> + case RTE_INTR_HANDLE_VFIO_REQ:
> + return -1;
> +#endif
> +#endif
> + /* not used at this moment */
> + case RTE_INTR_HANDLE_DEV_EVENT:
> + return -1;
> + /* unknown handle type */
> + default:
> + RTE_LOG(ERR, EAL,
> + "Unknown handle type of fd %d\n",
> + intr_handle->fd);
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +int
> rte_intr_disable(const struct rte_intr_handle *intr_handle)
> {
> if (intr_handle && intr_handle->type == RTE_INTR_HANDLE_VDEV)
> diff --git a/lib/librte_eal/rte_eal_version.map
> b/lib/librte_eal/rte_eal_version.map
> index 1892d9e..b3b2df4 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -309,6 +309,8 @@ DPDK_19.08 {
> rte_service_lcore_attr_reset_all;
> rte_service_may_be_active;
> rte_srand;
> + rte_intr_unmask;
> + rte_intr_mask;
>
> } DPDK_19.05;
>
> --
> 2.8.4
