On Fri, 4 Sep 2020 15:53:31 +0200
Björn Töpel <bjorn.to...@gmail.com> wrote:
> From: Björn Töpel <bjorn.to...@intel.com>
>
> Make the AF_XDP zero-copy path aware that the reason for redirect
> failure was due to full Rx queue. If so, exit the napi loop as soon as
> possible (exit the softirq processing), so that the userspace AF_XDP
> process can hopefully empty the Rx queue. This mainly helps the "one
> core scenario", where the userland process and Rx softirq processing
> is on the same core.
>
> Note that the early exit can only be performed if the "need wakeup"
> feature is enabled, because otherwise there is no notification
> mechanism available from the kernel side.
>
> This requires that the driver starts using the newly introduced
> xdp_do_redirect_ext() and xsk_do_redirect_rx_full() functions.
>
> Signed-off-by: Björn Töpel <bjorn.to...@intel.com>
> ---
> drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 23 ++++++++++++++------
> 1 file changed, 16 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
> b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
> index 3771857cf887..a4aebfd986b3 100644
> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
> @@ -93,9 +93,11 @@ int ixgbe_xsk_pool_setup(struct ixgbe_adapter *adapter,
>
> static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
> struct ixgbe_ring *rx_ring,
> - struct xdp_buff *xdp)
> + struct xdp_buff *xdp,
> + bool *early_exit)
> {
> int err, result = IXGBE_XDP_PASS;
> + enum bpf_map_type map_type;
> struct bpf_prog *xdp_prog;
> struct xdp_frame *xdpf;
> u32 act;
> @@ -116,8 +118,13 @@ static int ixgbe_run_xdp_zc(struct ixgbe_adapter
> *adapter,
> result = ixgbe_xmit_xdp_ring(adapter, xdpf);
> break;
> case XDP_REDIRECT:
> - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
> - result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED;
> + err = xdp_do_redirect_ext(rx_ring->netdev, xdp, xdp_prog,
> &map_type);
> + if (err) {
> + *early_exit = xsk_do_redirect_rx_full(err, map_type);
Have you tried calling xdp_do_flush (that calls __xsk_map_flush()) and
(I guess) xsk_set_rx_need_wakeup() here, instead of stopping the loop?
(Or doing this in xsk core).
Looking at the code, the AF_XDP frames are "published" in the queue
rather late for AF_XDP. Maybe in an orthogonal optimization, have you
considered "publishing" the ring producer when e.g. the queue is
half-full?
> + result = IXGBE_XDP_CONSUMED;
> + } else {
> + result = IXGBE_XDP_REDIR;
> + }
> break;
> default:
> bpf_warn_invalid_xdp_action(act);
> @@ -235,8 +242,8 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
> unsigned int total_rx_bytes = 0, total_rx_packets = 0;
> struct ixgbe_adapter *adapter = q_vector->adapter;
> u16 cleaned_count = ixgbe_desc_unused(rx_ring);
> + bool early_exit = false, failure = false;
> unsigned int xdp_res, xdp_xmit = 0;
> - bool failure = false;
> struct sk_buff *skb;
>
> while (likely(total_rx_packets < budget)) {
> @@ -288,7 +295,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
>
> bi->xdp->data_end = bi->xdp->data + size;
> xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool);
> - xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp);
> + xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp,
> &early_exit);
>
> if (xdp_res) {
> if (xdp_res & (IXGBE_XDP_TX | IXGBE_XDP_REDIR))
> @@ -302,6 +309,8 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
>
> cleaned_count++;
> ixgbe_inc_ntc(rx_ring);
> + if (early_exit)
> + break;
> continue;
> }
>
> @@ -346,12 +355,12 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector
> *q_vector,
> q_vector->rx.total_bytes += total_rx_bytes;
>
> if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) {
> - if (failure || rx_ring->next_to_clean == rx_ring->next_to_use)
> + if (early_exit || failure || rx_ring->next_to_clean ==
> rx_ring->next_to_use)
> xsk_set_rx_need_wakeup(rx_ring->xsk_pool);
> else
> xsk_clear_rx_need_wakeup(rx_ring->xsk_pool);
>
> - return (int)total_rx_packets;
> + return early_exit ? 0 : (int)total_rx_packets;
> }
> return failure ? budget : (int)total_rx_packets;
> }
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
