Hi Jun,

This patch and 47/65 are not in v5-S3. These comments are for when
they come back. I only read the code, I did not run v4-S3 on LX2160A.

My three main concerns:
    1. the PMD changes the CPU frequency governor;
    2. the Rx burst uses the lcore id as a Linux CPU number;
    3. one DPIO per Rx queue is expensive.

Details below.

On Fri, Oct 23, 2026 at 3:49 PM Prashant Gupta <[email protected]> wrote:
>
> From: Jun Yang <[email protected]>
>
> The DPAA2 rx-queue interrupt path never delivered a wakeup to the
> application.

This is not correct. The current code works if the application calls
rte_eth_dev_rx_intr_enable() before rte_eth_dev_rx_intr_ctl_q(). This
order is documented in dpaa2.rst, and grout uses it.

l3fwd-power calls them in the other order, so it does not work. I
sent a series to fix l3fwd-power:
https://mails.dpdk.org/archives/dev/2026-September/345474.html

Please reword the commit message. Also, what does a DPIO per queue
bring, once l3fwd-power is fixed?

[...]
> +RTE_EXPORT_INTERNAL_SYMBOL(dpaa2_dpio_intr_thread_affinity)
> +void dpaa2_dpio_intr_thread_affinity(struct dpaa2_dpio_dev *dpio_dev)
> +{
> +       int ret;
> +
> +       ret = dpaa2_dpio_intr_affinity_set(dpio_dev);
> +       if (ret) {
> +               DPAA2_BUS_WARN("Failed(%d) to affinity dpio.%d's interrupt on 
> core%d",
> +                       ret, dpio_dev->hw_id, dpio_dev->cpu_affine);
> +       }
> +       ret = dpaa2_dpio_configure_stashing(dpio_dev);
> +       if (ret) {
> +               DPAA2_BUS_WARN("Failed(%d) to configure dpio.%d's stashing on 
> core%d",
> +                       ret, dpio_dev->hw_id, dpio_dev->cpu_affine);
> +       };
> +
> +       ret = dpaa2_dpio_cpufreq_governor(dpio_dev->cpu_affine, false);

Please remove the governor change. The power policy belongs to the
application, not to the PMD.

grout uses the schedutil governor. The CPU runs fast while the worker
polls, and slows down while the worker sleeps on the Rx interrupt.
If the PMD forces "performance", the CPU never slows down, and Rx
interrupts save no power.

The restore is also wrong in three cases:
    - Two ports are polled by the same lcore. Closing the first port
      restores the governor, but the second port is still running.
    - A queue moves from CPU A to CPU B. Only B is restored. A stays
      in "performance".
    - A queue was set up but never polled. Its cpu_affine is still -1,
      so the restore applies to all CPUs.

[...]
> @@ -1579,17 +1582,11 @@ dpaa2_dev_rx_queue_setup(struct rte_eth_dev *dev,
>         dpaa2_q->bp_array = rte_dpaa2_bpid_info;
>         dpaa2_q->offloads = rx_conf->offloads;
>
> -       /* NAPI: grab a DPCON channel so dev_start can bind this FQ 
> statically.
> -        * The DQRR burst replaces the poll path for every queue at once, so a
> -        * missing channel is fatal rather than a silent per-queue fallback.
> -        */
> -       rte_atomic_store_explicit(&dpaa2_q->napi_sub_dpio, NULL,
> -                       rte_memory_order_release);
> -       if (dev->data->dev_conf.intr_conf.rxq && !dpaa2_q->napi_dpcon) {
> -               dpaa2_q->napi_dpcon = rte_dpaa2_alloc_dpcon_dev();
> +       if (dev->data->dev_conf.intr_conf.rxq) {
> +               if (!dpaa2_q->napi_dpcon)
> +                       dpaa2_q->napi_dpcon = rte_dpaa2_alloc_dpcon_dev();
>                 if (!dpaa2_q->napi_dpcon) {
> -                       DPAA2_PMD_ERR("rxq %d: no DPCON for rx-queue 
> interrupts",
> -                               rx_queue_id);
> +                       DPAA2_PMD_ERR("rxq %d: no DPCON for rx-queue 
> interrupts", rx_queue_id);
>                         ret = -ENODEV;
>                         goto err_quit;
>                 }
> @@ -1600,6 +1597,27 @@ dpaa2_dev_rx_queue_setup(struct rte_eth_dev *dev,
>                         DPAA2_PMD_ERR("rxq %d: dpcon_enable: %d", 
> rx_queue_id, ret);
>                         goto err_quit;
>                 }
> +               if (!dpaa2_q->napi_dpio) {
> +                       dpaa2_q->napi_dpio = rte_dpaa2_alloc_dpio_device(true,
> +                               DPAA2_PORTAL_INTR_COALESC_DISABLE, -1, false);

Each Rx queue now takes its own DPIO.

For example, with 12 workers, each polling one Rx queue per port:
    - before this patch: 12 DPIOs for Rx (one per worker, shared by
      all its queues)
    - with this patch:   24 DPIOs for Rx with 2 ports, 48 with 4 ports

  How many DPIOs can each SoC provide (LX2160A, LS2088A, LS1088A)?

[...]
> @@ -3706,74 +3699,42 @@ dpaa2_dev_rx_queue_intr_enable(struct rte_eth_dev 
> *dev, uint16_t queue_id)
>  {
>         struct dpaa2_dev_priv *priv = dev->data->dev_private;
>         struct dpaa2_queue *dpaa2_q = priv->rx_vq[queue_id];
> -       struct dpaa2_dpio_dev *dpio, *old;
>         int ret;
>
> -       /* the generic API has no dev_started guard (unlike the burst dummy 
> ops),
> -        * so it can reach us after a stop; do not arm a stopped port, let the
> -        * caller fall back to polling.
> -        */
> -       if (!dev->data->dev_started)
> -               return -EIO;

Why is this check removed?

The ethdev layer does not check that the port is started, so the
driver now arms a queue on a stopped port.

> -       if (!dpaa2_q->napi_dpcon)
> -               return -ENOTSUP;        /* no channel -> caller keeps polling 
> */
> -
> -       if (dpaa2_affine_qbman_ethrx_swp())
> -               return -EIO;
> -       dpio = DPAA2_PER_LCORE_ETHRX_DPIO;
> +       if (dpaa2_q->napi_armed)
> +               return 0;
>
> -       old = rte_atomic_load_explicit(&dpaa2_q->napi_sub_dpio, 
> rte_memory_order_acquire);
> -       if (old && old != dpio && dpaa2_q->napi_armed) {
> -               DPAA2_PMD_ERR("rxq %d still armed on another portal; disable 
> it first",
> -                             queue_id);
> -               return -EBUSY;
> -       }
> +       if (!dpaa2_q->napi_dpcon || !dpaa2_q->napi_dpio)
> +               return -ENOTSUP;
>
>         ret = dpaa2_napi_quiesce_vdq(dpaa2_q);
>         if (ret)
> -               return ret;
> +               goto err_quit;
>
> -       /* holdoff 0: the DQRI fires immediately on the first entry, so the
> -        * threshold is moot (1 here). build_epoll=false: the generic rx-intr 
> API
> -        * waits on the application epoll, not the portal's private one. 
> Idempotent
> -        * per dpio.
> -        */
> -       ret = dpaa2_dpio_intr_init(dpio, 1, 0, false);
> +       ret = dpaa2_napi_subscribe(dev, queue_id, dpaa2_q->napi_dpio);

Before this patch, dpaa2_napi_subscribe() was called only when the
portal changed.
Now it is called every time the queue is armed, so every time the
worker goes to sleep.

Each call sends an MC command (dpcon_set_notification()).

The DPIO never changes now. Please call it once, in rx_queue_setup().

[...]
> diff --git a/drivers/net/dpaa2/dpaa2_rxtx.c b/drivers/net/dpaa2/dpaa2_rxtx.c
> index 0df758148c..efaa862623 100644
> --- a/drivers/net/dpaa2/dpaa2_rxtx.c
> +++ b/drivers/net/dpaa2/dpaa2_rxtx.c
> @@ -963,7 +963,7 @@ dpaa2_dev_prefetch_rx_common(struct dpaa2_queue *dpaa2_q,
>  {
>         /* Function receive frames for a given device and VQ*/
>         struct qbman_result *dq_storage, *dq_storage1 = NULL;
> -       int ret, num_rx = 0;
> +       int ret, num_rx = 0, cpu = rte_lcore_id();
>         bool by_channel = dpaa2_q->napi_dpcon ? true : false;
>         uint32_t id, pull_size;
>         uint8_t pending, status;
> @@ -971,26 +971,36 @@ dpaa2_dev_prefetch_rx_common(struct dpaa2_queue 
> *dpaa2_q,
>         const struct qbman_fd *fd;
>         struct qbman_pull_desc pulldesc;
>         struct queue_storage_info_t *q_storage;
> +       struct dpaa2_dpio_dev *dpio;
>
> -       q_storage = dpaa2_q->q_storage[rte_lcore_id()];
> +       q_storage = dpaa2_q->q_storage[cpu];
>
> -       if (unlikely(!DPAA2_PER_LCORE_ETHRX_DPIO)) {
> -               ret = dpaa2_affine_qbman_ethrx_swp();
> -               if (ret) {
> -                       DPAA2_PMD_ERR("Failure in affining portal");
> -                       return 0;
> +       if (dpaa2_q->napi_dpio) {
> +               dpio = dpaa2_q->napi_dpio;
> +               if (unlikely(dpio->cpu_affine != cpu)) {
> +                       dpio->cpu_affine = cpu;
> +                       dpaa2_dpio_intr_thread_affinity(dpio);

Two problems here.

First, "cpu" is a DPDK lcore id, not a Linux CPU number. They are
equal only with the default mapping. They differ with --lcores, and
for threads created with rte_thread_register(), like grout workers.
In that case the IRQ affinity, the stashing destination and the
governor are set for the wrong CPU. Please use dpaa2_get_core_id(),
as the removed dpaa2_dpio_affine_intr_to_core() did.

Second, this runs inside rte_eth_rx_burst(). cpu_affine starts at -1,
so it runs on the first burst of every queue, and again each
time the queue moves. It reads /proc/interrupts, writes
/proc/irq/N/smp_affinity_list and sends an MC command.

 Proposal:
    - set the IRQ affinity in dpaa2_dev_rx_queue_intr_enable(). It
      runs on the polling lcore, just before the worker sleeps.
    - keep only the stashing update in the burst.

>                 }
> +       } else {
> +               if (unlikely(!DPAA2_PER_LCORE_ETHRX_DPIO)) {
> +                       ret = dpaa2_affine_qbman_ethrx_swp();
> +                       if (ret) {
> +                               DPAA2_PMD_ERR("Failure in affining portal");
> +                               return 0;
> +                       }
> +               }
> +               dpio = DPAA2_PER_LCORE_ETHRX_DPIO;
>         }
> +       swp = dpio->sw_portal;
>
> -       swp = DPAA2_PER_LCORE_ETHRX_PORTAL;
>         if (dpaa2_q->napi_dpcon)
>                 id = dpaa2_q->napi_dpcon->qbman_ch_id;
>         else
>                 id = dpaa2_q->fqid;
>         pull_size = (nb_pkts > dpaa2_dqrr_size) ? dpaa2_dqrr_size : nb_pkts;
>         if (unlikely(!q_storage->active_dqs)) {
> -               dpaa2_dev_rx_active_dq_storage(q_storage, 
> DPAA2_PER_LCORE_ETHRX_DPIO,
> -                       id, pull_size, by_channel, qbman_pull_type_active);
> +               dpaa2_dev_rx_active_dq_storage(q_storage, dpio, id, pull_size,
> +                       by_channel, qbman_pull_type_active);
>         }
>
>         dq_storage = q_storage->active_dqs;

The DPIO never changes now. Please call it once, in
rx_queue_setup().The DPIO never changes now. Please call it once, in
rx_queue_setup().The DPIO never changes now. Please call it once, in
rx_queue_setup().A question about queue migration. grout moves a queue
from one worker
to another without stopping the port.

When the queue moves from lcore A to lcore B, the DPIO follows the
queue, but q_storage is still per lcore. If A left a VDQ in flight,
its frames stay in q_storage[A]. B waits for that VDQ to complete,
then pulls into q_storage[B]. It never reads the frames in
q_storage[A].

This is not new: q_storage is already per lcore in 26.07, and the
only cleanup, dpaa2_clear_queue_active_dps(), runs at close and does
not read the frames. But now that the DPIO belongs to the queue,
q_storage could belong to the queue too. Then B would simply read
the frames left by A. What do you think?

Last point: dpaa2.rst and the comments in
dpaa2_dev_rx_queue_intr_enable()/disable() still describe the old
model (shared per-lcore portal, fd bound when the queue is armed,
sibling queues).

dpaa2.rst should also say that each Rx queue needs a DPIO, not only a DPCON.

Thanks,

Maxime

Reply via email to