[dpdk-dev] [PATCH v2] net/mlx5: fix init on secondary process

[Olivier Gournet]

Fixes: 1d88ba171942 ("net/mlx5: refactor Tx data path")
Fixes: 21c8bb4928c9 ("net/mlx5: split Tx queue structure")

Signed-off-by: Olivier Gournet 
---
 drivers/net/mlx5/mlx5_ethdev.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index c76e754..188ade9 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -1309,11 +1309,13 @@ mlx5_secondary_data_setup(struct priv *priv)
continue;
primary_txq_ctrl = container_of(primary_txq,
struct txq_ctrl, txq);
-   txq_ctrl = rte_calloc_socket("TXQ", 1, sizeof(*txq_ctrl), 0,
+   txq_ctrl = rte_calloc_socket("TXQ", 1, sizeof(*txq_ctrl) +
+(1 << primary_txq->elts_n) *
+sizeof(struct rte_mbuf *), 0,
 primary_txq_ctrl->socket);
if (txq_ctrl != NULL) {
if (txq_ctrl_setup(priv->dev,
-  primary_txq_ctrl,
+  txq_ctrl,
   1 << primary_txq->elts_n,
   primary_txq_ctrl->socket,
   NULL) == 0) {
-- 
2.1.4

[dpdk-dev] [PATCH] net/mlx5: fix init on secondary process

[Olivier Gournet]

Fixes: 1d88ba171942 ("net/mlx5: refactor Tx data path")

Signed-off-by: Olivier Gournet 
---
 drivers/net/mlx5/mlx5_ethdev.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index 130e15d..6f39965 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -1308,11 +1308,13 @@ mlx5_secondary_data_setup(struct priv *priv)
continue;
primary_txq_ctrl = container_of(primary_txq,
struct txq_ctrl, txq);
-   txq_ctrl = rte_calloc_socket("TXQ", 1, sizeof(*txq_ctrl), 0,
+   txq_ctrl = rte_calloc_socket("TXQ", 1, sizeof(*txq_ctrl) +
+primary_txq->elts_n *
+sizeof(struct rte_mbuf *), 0,
 primary_txq_ctrl->socket);
if (txq_ctrl != NULL) {
if (txq_ctrl_setup(priv->dev,
-  primary_txq_ctrl,
+  txq_ctrl,
   primary_txq->elts_n,
   primary_txq_ctrl->socket,
   NULL) == 0) {
-- 
2.1.4

[dpdk-dev] [PATCH] net/mlx5: fix socket field initialization

[Olivier Gournet]

Fixes: a1366b1a2be (net/mlx5: add reference counter on DPDK Rx queues)

Signed-off-by: Olivier Gournet 
---
 drivers/net/mlx5/mlx5_rxq.c | 1 +
 drivers/net/mlx5/mlx5_txq.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index a1f382b..53bb423 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -894,6 +894,7 @@ mlx5_priv_rxq_new(struct priv *priv, uint16_t idx, uint16_t 
desc,
 0, socket);
if (!tmpl)
return NULL;
+   tmpl->socket = socket;
if (priv->dev->data->dev_conf.intr_conf.rxq)
tmpl->irq = 1;
/* Enable scattered packets support for this queue if necessary. */
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index fbb2630..9c5860f 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -580,6 +580,7 @@ mlx5_priv_txq_new(struct priv *priv, uint16_t idx, uint16_t 
desc,
assert(desc > MLX5_TX_COMP_THRESH);
tmpl->txq.flags = conf->txq_flags;
tmpl->priv = priv;
+   tmpl->socket = socket;
tmpl->txq.elts_n = log2above(desc);
if (priv->mps == MLX5_MPW_ENHANCED)
tmpl->txq.mpw_hdr_dseg = priv->mpw_hdr_dseg;
-- 
2.1.4

[dpdk-dev] mlx5 and secondary processes

[Olivier Gournet]

Hi,

I can't get TX on secondary process to works with the lastest dpdk, it was 
running fine with dpdk-16.11.
RX/TX is ok on primary processs, and I'm not interested in RX on secondary 
process. Each process has its
owns TX queues.

I upgraded everything to:
  linux 4.14
  dpdk 17.11-rc4
  rdma-core from yesterday git-master
  mlx-fw 12.21.1000

On seconday process, TX queue gets full and is never emptied.
It seems like bf_reg is correctly re-mapped from secondary process; here's 
dmesg from primary process:

[...]
[85361.895778] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x6, pfn 
0x000f8020
[85361.895781] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 
0x77faa000, PA 0xf802
[85361.895784] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x7, pfn 
0x000f8021
[85361.895787] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 
0x77fa9000, PA 0xf8021000

Then from secondary process:

[...]
[85408.038295] mlx5_1:mlx5_ib_mmap:1778:(pid 9551): mapped internal timer at 
0x77f7f000, PA 0xf8001000
[85408.040229] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x6, pfn 
0x000f8020
[85408.040233] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 
0x77faa000, PA 0xf802
[85408.040239] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x7, pfn 
0x000f8021
[85408.040241] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 
0x77fa9000, PA 0xf8021000

But then from mlx5_rxtx.h:mlx5_tx_dbrec_cond_wmb() it doesn't seems like writing 
to txq->bf_reg has any effect.
I know it isn't supposed to be read, but when reading from primary process is 
always give *dst==0xe5ccdabae5ccdaba,
whereas from secondary process *dst is zero or the last written value.

I don't really have any other clues, and don't know where to search. Can 
anybody give me some hints ?

Regards,


=

# ibv_devinfo -d mlx5_1
hca_id: mlx5_1
transport:  InfiniBand (0)
fw_ver: 12.21.1000
node_guid:  248a:0703:008a:638e
sys_image_guid: 248a:0703:008a:638e
vendor_id:  0x02c9
vendor_part_id: 4115
hw_ver: 0x0
board_id:   MT_2140110033
phys_port_cnt:  1
Device ports:
port:   1
state:  PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid:   0
port_lmc:   0x00
link_layer: Ethernet

Re: [dpdk-dev] mlx5 and secondary processes

[Olivier Gournet]

Hi Nelio,

On 11/14/2017 11:34 AM, Nelio Laranjeiro wrote:

Hi Olivier,

On Tue, Nov 14, 2017 at 10:58:02AM +0100, Olivier Gournet wrote:

Hi,

I can't get TX on secondary process to works with the lastest dpdk, it was 
running fine with dpdk-16.11.
RX/TX is ok on primary processs, and I'm not interested in RX on secondary 
process. Each process has its
owns TX queues.

I upgraded everything to:
   linux 4.14
   dpdk 17.11-rc4
   rdma-core from yesterday git-master
   mlx-fw 12.21.1000

On seconday process, TX queue gets full and is never emptied.
It seems like bf_reg is correctly re-mapped from secondary process; here's 
dmesg from primary process:

[...]
[85361.895778] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x6, pfn 
0x000f8020
[85361.895781] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 
0x77faa000, PA 0xf802
[85361.895784] mlx5_1:uar_mmap:1722:(pid 9533): uar idx 0x7, pfn 
0x000f8021
[85361.895787] mlx5_1:uar_mmap:1735:(pid 9533): mapped best effort WC at 
0x77fa9000, PA 0xf8021000

Then from secondary process:

[...]
[85408.038295] mlx5_1:mlx5_ib_mmap:1778:(pid 9551): mapped internal timer at 
0x77f7f000, PA 0xf8001000
[85408.040229] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x6, pfn 
0x000f8020
[85408.040233] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 
0x77faa000, PA 0xf802
[85408.040239] mlx5_1:uar_mmap:1722:(pid 9551): uar idx 0x7, pfn 
0x000f8021
[85408.040241] mlx5_1:uar_mmap:1735:(pid 9551): mapped best effort WC at 
0x77fa9000, PA 0xf8021000

But then from mlx5_rxtx.h:mlx5_tx_dbrec_cond_wmb() it doesn't seems like writing 
to txq->bf_reg has any effect.
I know it isn't supposed to be read, but when reading from primary process is 
always give *dst==0xe5ccdabae5ccdaba,
whereas from secondary process *dst is zero or the last written value.

I don't really have any other clues, and don't know where to search. Can 
anybody give me some hints ?


Since Yongseok patch [1] the UAR pages are no more configured though the
Write Combining of the PCI but in your logs it seems it is still using
the cache.

Do you have the "MLX5_SHUT_UP_BF" set to 1 in your environment?  If not
does adding it solves your issue?


I set it (on both primary and secondary) and it is seen by kernel driver,
but unfortunately it is still not working.

[98321.605791] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x77faa000, PA 
0xf802
[98321.605799] mlx5_1:uar_mmap:1735:(pid 9706): mapped NC at 0x77fa9000, PA 
0xf8021000

Thanks,