[dpdk-dev] [PATCH v2 00/15] Add support for secondary queue set in nicvf thunderx driver
This series of patches adds support for secondary queue set in nicvf thunderx driver There are two types of VFs: - Primary VF - Secondary VF Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. In case port is configured to use more than 8 queues, then it requires one (or more) secondary VF. Each secondary VF adds additional 8 queues to the queue set. During PMD driver initialization, the primary VF's are enumerated by checking the specific flag (see READY message). They are at the beginning of VF list (the remain ones are secondary VF's). The primary VFs are used as master queue sets. Secondary VFs provide additional queue sets for primary ones. If a port is configured for more then 8 queues then it will request for additional queues from secondary VFs. Secondary VFs cannot be shared between primary VFs. Primary VFs are present on the tail of the 'Network devices using kernel driver' list, secondary VFs are on the remaining tail of the list. The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attach with an id below or equal to this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. This patchset also contains other cleanups and improvements like fixing erroneous checksum calculation and preparing the thunderx driver for the multi queue set feature support. These changes base on the following pending patches: [dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver http://dpdk.org/dev/patchwork/patch/14963/ [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag http://dpdk.org/dev/patchwork/patch/14964/ [dpdk-dev,3/3] net/thunderx: add 81xx SoC support http://dpdk.org/dev/patchwork/patch/14965/ Kamil Rytarowski (15): net/thunderx: cleanup the driver before adding new features net/thunderx: correct transmit checksum handling net/thunderx/base: add family of functions to store qsets net/thunderx/base: add secondary queue set support net/thunderx: add family of functions to store DPDK qsets net/thunderx: add secondary queue set in interrupt functions net/thunderx: remove problematic private_data->eth_dev link net/thunderx: add helper utils for secondary qset support net/thunderx: add secondary qset support in dev stop/close net/thunderx: add secondary qset support in device start net/thunderx: add secondary qset support in device configure net/thunderx: add final bits for secondary queue support net/thunderx: document secondary queue set support ethdev: Support VFs on the different PCI domains net/thunderx: Bump driver version to 2.0 doc/guides/nics/thunderx.rst | 148 +- drivers/net/thunderx/Makefile | 2 + drivers/net/thunderx/base/nicvf_bsvf.c| 72 +++ drivers/net/thunderx/base/nicvf_bsvf.h| 76 +++ drivers/net/thunderx/base/nicvf_hw.c | 10 +- drivers/net/thunderx/base/nicvf_hw.h | 6 +- drivers/net/thunderx/base/nicvf_hw_defs.h | 1 + drivers/net/thunderx/base/nicvf_mbox.c| 47 +- drivers/net/thunderx/base/nicvf_mbox.h| 21 +- drivers/net/thunderx/nicvf_ethdev.c | 780 +- drivers/net/thunderx/nicvf_ethdev.h | 41 +- drivers/net/thunderx/nicvf_rxtx.c | 14 +- drivers/net/thunderx/nicvf_struct.h | 6 +- drivers/net/thunderx/nicvf_svf.c | 78 +++ drivers/net/thunderx/nicvf_svf.h | 66 +++ lib/librte_ether/rte_ethdev.c | 2 +- 16 files changed, 1097 insertions(+), 273 deletions(-) create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h create mode 100644 drivers/net/thunderx/nicvf_svf.c create mode 100644 drivers/net/thunderx/nicvf_svf.h -- 1.9.1
[dpdk-dev] [PATCH v2 01/15] net/thunderx: cleanup the driver before adding new features
From: Kamil Rytarowski
Refactored features:
- enable nicvf_qset_rbdr_precharge to handle secondary queue sets
- rte_free already handles NULL pointer
- check mempool flags to predict being contiguous in memory
- prohibit to use mempool with multiple memory chunks
- simplify local construct of accessing nb_rx_queues
- enable NICVF_CAP_CQE_RX2 on CN88XX PASS2.0 hardware.
- remove redundant check for RSS size in nicvf_eth_dev_init
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/base/nicvf_hw.c | 10 +-
drivers/net/thunderx/base/nicvf_hw.h | 6 +++---
drivers/net/thunderx/nicvf_ethdev.c | 37 ++--
3 files changed, 26 insertions(+), 27 deletions(-)
diff --git a/drivers/net/thunderx/base/nicvf_hw.c
b/drivers/net/thunderx/base/nicvf_hw.c
index 4bdd183..1f08ef2 100644
--- a/drivers/net/thunderx/base/nicvf_hw.c
+++ b/drivers/net/thunderx/base/nicvf_hw.c
@@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
return NICVF_ERR_BASE_INIT;
if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
- nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
+ nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
@@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
}
int
-nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
- rbdr_pool_get_handler handler,
- void *opaque, uint32_t max_buffs)
+nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+ uint16_t ridx, rbdr_pool_get_handler handler,
+ uint32_t max_buffs)
{
struct rbdr_entry_t *desc, *desc0;
struct nicvf_rbdr *rbdr = nic->rbdr;
@@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
if (count >= max_buffs)
break;
desc0 = desc + count;
- phy = handler(opaque);
+ phy = handler(dev, nic);
if (phy) {
desc0->full_addr = phy;
count++;
diff --git a/drivers/net/thunderx/base/nicvf_hw.h
b/drivers/net/thunderx/base/nicvf_hw.h
index a6cda82..2b8738b 100644
--- a/drivers/net/thunderx/base/nicvf_hw.h
+++ b/drivers/net/thunderx/base/nicvf_hw.h
@@ -85,7 +85,7 @@ enum nicvf_err_e {
NICVF_ERR_RSS_GET_SZ,/* -8171 */
};
-typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
+typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
struct nicvf_hw_rx_qstats {
uint64_t q_rx_bytes;
@@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
-int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
- rbdr_pool_get_handler handler, void *opaque,
+int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+ uint16_t ridx, rbdr_pool_get_handler handler,
uint32_t max_buffs);
int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 914035e..3cf32bf 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
int ret;
/* Count started rx queues */
- for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+ for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
if (dev->data->rx_queue_state[qidx] ==
RTE_ETH_QUEUE_STATE_STARTED)
qcnt++;
@@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t
qidx)
static void
nicvf_dev_rx_queue_release(void *rx_queue)
{
- struct nicvf_rxq *rxq = rx_queue;
-
PMD_INIT_FUNC_TRACE();
- if (rxq)
- rte_free(rxq);
+ rte_free(rx_queue);
}
static int
@@ -1070,6 +1067,7 @@ nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t
qidx)
return nicvf_stop_tx_queue(dev, qidx);
}
+
static int
nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t qidx,
uint16_t nb_desc, unsigned int socket_id,
@@ -1087,9 +1085,15 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev,
uint16_t qidx,
PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
socket_id, nic->node);
- /* Mempool memory
[dpdk-dev] [PATCH v2 02/15] net/thunderx: correct transmit checksum handling
From: Kamil Rytarowski
The symbols PKT_TX_TCP_CKSUM and PKT_TX_UDP_CKSUM are not bits on a
bitmask. Set l3_offset always for TX offloads, not just for PKT_TX_IP_CKSUM
being true.
Fixes: 1c421f18e095 ("net/thunderx: add single and multi-segment Tx")
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_rxtx.c | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_rxtx.c
b/drivers/net/thunderx/nicvf_rxtx.c
index eb51a72..e15c730 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -70,19 +70,20 @@ fill_sq_desc_header(union sq_entry_t *entry, struct
rte_mbuf *pkt)
ol_flags = pkt->ol_flags & NICVF_TX_OFFLOAD_MASK;
if (unlikely(ol_flags)) {
/* L4 cksum */
- if (ol_flags & PKT_TX_TCP_CKSUM)
+ uint64_t l4_flags = ol_flags & PKT_TX_L4_MASK;
+ if (l4_flags == PKT_TX_TCP_CKSUM)
sqe.hdr.csum_l4 = SEND_L4_CSUM_TCP;
- else if (ol_flags & PKT_TX_UDP_CKSUM)
+ else if (l4_flags == PKT_TX_UDP_CKSUM)
sqe.hdr.csum_l4 = SEND_L4_CSUM_UDP;
else
sqe.hdr.csum_l4 = SEND_L4_CSUM_DISABLE;
+
+ sqe.hdr.l3_offset = pkt->l2_len;
sqe.hdr.l4_offset = pkt->l3_len + pkt->l2_len;
/* L3 cksum */
- if (ol_flags & PKT_TX_IP_CKSUM) {
+ if (ol_flags & PKT_TX_IP_CKSUM)
sqe.hdr.csum_l3 = 1;
- sqe.hdr.l3_offset = pkt->l2_len;
- }
}
entry->buff[0] = sqe.buff[0];
--
1.9.1
[dpdk-dev] [PATCH v2 03/15] net/thunderx/base: add family of functions to store qsets
From: Kamil Rytarowski
This interface (nicvf_bsvf) will be used for secondary queue set support.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/Makefile | 1 +
drivers/net/thunderx/base/nicvf_bsvf.c | 72
drivers/net/thunderx/base/nicvf_bsvf.h | 76 ++
3 files changed, 149 insertions(+)
create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 8ea6b45..711400a 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -57,6 +57,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_rxtx.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.c
b/drivers/net/thunderx/base/nicvf_bsvf.c
new file mode 100644
index 000..9e028a3
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.c
@@ -0,0 +1,72 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include
+#include
+#include
+
+#include "nicvf_bsvf.h"
+#include "nicvf_plat.h"
+
+static SIMPLEQ_HEAD(, svf_entry) head = SIMPLEQ_HEAD_INITIALIZER(head);
+
+void
+nicvf_bsvf_push(struct svf_entry *entry)
+{
+ assert(entry != NULL);
+ assert(entry->vf != NULL);
+
+ SIMPLEQ_INSERT_TAIL(&head, entry, next);
+}
+
+struct svf_entry *
+nicvf_bsvf_pop(void)
+{
+ struct svf_entry *entry;
+
+ assert(!SIMPLEQ_EMPTY(&head));
+
+ entry = SIMPLEQ_FIRST(&head);
+
+ assert(entry != NULL);
+ assert(entry->vf != NULL);
+
+ SIMPLEQ_REMOVE_HEAD(&head, next);
+
+ return entry;
+}
+
+int
+nicvf_bsvf_empty(void)
+{
+ return SIMPLEQ_EMPTY(&head);
+}
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.h
b/drivers/net/thunderx/base/nicvf_bsvf.h
new file mode 100644
index 000..5d5a25e
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.h
@@ -0,0 +1,76 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE
[dpdk-dev] [PATCH v2 04/15] net/thunderx/base: add secondary queue set support
From: Kamil Rytarowski
Changes:
- add new message sqs_alloc in mailbox
- add a queue container to hold secondary qsets.
- add nicvf_mbox_request_sqs
- handle new mailbox messages for secondary queue set support
- register secondary queue sets for furthe reuse
- register the number secondary queue sets in MSG_QS_CFG
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/base/nicvf_hw_defs.h | 1 +
drivers/net/thunderx/base/nicvf_mbox.c| 47 ++-
drivers/net/thunderx/base/nicvf_mbox.h| 21 --
drivers/net/thunderx/nicvf_struct.h | 5
4 files changed, 71 insertions(+), 3 deletions(-)
diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h
b/drivers/net/thunderx/base/nicvf_hw_defs.h
index 2f2b225..e144d44 100644
--- a/drivers/net/thunderx/base/nicvf_hw_defs.h
+++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
@@ -207,6 +207,7 @@
#define NICVF_CQE_RX2_RBPTR_WORD(7)
#define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
+#define assert_primary(nic) assert((nic)->sqs_mode == 0)
typedef uint64_t nicvf_phys_addr_t;
diff --git a/drivers/net/thunderx/base/nicvf_mbox.c
b/drivers/net/thunderx/base/nicvf_mbox.c
index 9c5cd83..3b7b8a5 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.c
+++ b/drivers/net/thunderx/base/nicvf_mbox.c
@@ -62,6 +62,9 @@ static const char *mbox_message[NIC_MBOX_MSG_MAX] = {
[NIC_MBOX_MSG_RESET_STAT_COUNTER] = "NIC_MBOX_MSG_RESET_STAT_COUNTER",
[NIC_MBOX_MSG_CFG_DONE] = "NIC_MBOX_MSG_CFG_DONE",
[NIC_MBOX_MSG_SHUTDOWN] = "NIC_MBOX_MSG_SHUTDOWN",
+ [NIC_MBOX_MSG_RES_BIT]= "NIC_MBOX_MSG_RES_BIT",
+ [NIC_MBOX_MSG_RSS_SIZE_RES_BIT] = "NIC_MBOX_MSG_RSS_SIZE",
+ [NIC_MBOX_MSG_ALLOC_SQS_RES_BIT] = "NIC_MBOX_MSG_ALLOC_SQS",
};
static inline const char * __attribute__((unused))
@@ -173,7 +176,7 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
case NIC_MBOX_MSG_NACK:
nic->pf_nacked = true;
break;
- case NIC_MBOX_MSG_RSS_SIZE:
+ case NIC_MBOX_MSG_RSS_SIZE_RES_BIT:
nic->rss_info.rss_size = mbx.rss_size.ind_tbl_size;
nic->pf_acked = true;
break;
@@ -183,6 +186,26 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
nic->speed = mbx.link_status.speed;
nic->pf_acked = true;
break;
+ case NIC_MBOX_MSG_ALLOC_SQS_RES_BIT:
+ assert_primary(nic);
+ if (mbx.sqs_alloc.qs_count != nic->sqs_count) {
+ nicvf_log_error("Received %" PRIu8 "/%" PRIu8
+ " secondary qsets",
+ mbx.sqs_alloc.qs_count,
+ nic->sqs_count);
+ abort();
+ }
+ for (i = 0; i < mbx.sqs_alloc.qs_count; i++) {
+ if (mbx.sqs_alloc.svf[i] != nic->snicvf[i]->vf_id) {
+ nicvf_log_error("Received secondary qset[%zu] "
+ "ID %" PRIu8 " expected %"
+ PRIu8, i, mbx.sqs_alloc.svf[i],
+ nic->snicvf[i]->vf_id);
+ abort();
+ }
+ }
+ nic->pf_acked = true;
+ break;
default:
nicvf_log_error("Invalid message from PF, msg_id=0x%hhx %s",
mbx.msg.msg, nicvf_mbox_msg_str(mbx.msg.msg));
@@ -314,11 +337,33 @@ nicvf_mbox_qset_config(struct nicvf *nic, struct
pf_qs_cfg *qs_cfg)
/* Send a mailbox msg to PF to config Qset */
mbx.msg.msg = NIC_MBOX_MSG_QS_CFG;
mbx.qs.num = nic->vf_id;
+ mbx.qs.sqs_count = nic->sqs_count;
mbx.qs.cfg = qs_cfg->value;
return nicvf_mbox_send_msg_to_pf(nic, &mbx);
}
int
+nicvf_mbox_request_sqs(struct nicvf *nic)
+{
+ struct nic_mbx mbx = { .msg = { 0 } };
+ size_t i;
+
+ assert_primary(nic);
+ assert(nic->sqs_count > 0);
+ assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+ mbx.sqs_alloc.msg = NIC_MBOX_MSG_ALLOC_SQS;
+ mbx.sqs_alloc.spec = 1;
+ mbx.sqs_alloc.qs_count = nic->sqs_count;
+
+ /* Set no of Rx/Tx queues in each of the SQsets */
+ for (i = 0; i < nic->sqs_count; i++)
+ mbx.sqs_alloc.svf[i] = nic->snicvf[i]->vf_id;
+
+ return nicvf_mbox_send_msg_to_pf(nic, &mbx);
+}
+
+int
nicvf_mbox_rq_drop_config(struct nicvf *nic,
[dpdk-dev] [PATCH v2 05/15] net/thunderx: add family of functions to store DPDK qsets
From: Kamil Rytarowski
These functions (nicvf_svf) are DPDK specialization of base/nicvf_bsvf.[ch]
ones.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/Makefile| 1 +
drivers/net/thunderx/nicvf_svf.c | 78
drivers/net/thunderx/nicvf_svf.h | 66 ++
3 files changed, 145 insertions(+)
create mode 100644 drivers/net/thunderx/nicvf_svf.c
create mode 100644 drivers/net/thunderx/nicvf_svf.h
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 711400a..bcab5f9 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -58,6 +58,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_svf.c
ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/nicvf_svf.c b/drivers/net/thunderx/nicvf_svf.c
new file mode 100644
index 000..f746e94
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.c
@@ -0,0 +1,78 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include
+#include
+
+#include
+#include
+
+#include "base/nicvf_bsvf.h"
+
+#include "nicvf_svf.h"
+
+void
+nicvf_svf_push(struct nicvf *vf)
+{
+ struct svf_entry *entry = NULL;
+
+ assert(vf != NULL);
+
+ entry = rte_zmalloc("nicvf", sizeof(*entry), RTE_CACHE_LINE_SIZE);
+ if (entry == NULL)
+ rte_panic("Cannoc allocate memory for svf_entry\n");
+
+ entry->vf = vf;
+
+ nicvf_bsvf_push(entry);
+}
+
+struct nicvf *
+nicvf_svf_pop(void)
+{
+ struct nicvf *vf;
+ struct svf_entry *entry;
+
+ entry = nicvf_bsvf_pop();
+
+ vf = entry->vf;
+
+ rte_free(entry);
+
+ return vf;
+}
+
+int
+nicvf_svf_empty(void)
+{
+ return nicvf_bsvf_empty();
+}
diff --git a/drivers/net/thunderx/nicvf_svf.h b/drivers/net/thunderx/nicvf_svf.h
new file mode 100644
index 000..6471aa5
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.h
@@ -0,0 +1,66 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS A
[dpdk-dev] [PATCH v2 06/15] net/thunderx: add secondary queue set in interrupt functions
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 41 ++---
1 file changed, 24 insertions(+), 17 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 3cf32bf..72e6667 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -101,31 +101,40 @@ nicvf_set_eth_link_status(struct nicvf *nic, struct
rte_eth_link *link)
static void
nicvf_interrupt(void *arg)
{
- struct nicvf *nic = arg;
+ struct rte_eth_dev *dev = arg;
+ struct nicvf *nic = nicvf_pmd_priv(dev);
if (nicvf_reg_poll_interrupts(nic) == NIC_MBOX_MSG_BGX_LINK_CHANGE) {
- if (nic->eth_dev->data->dev_conf.intr_conf.lsc)
- nicvf_set_eth_link_status(nic,
- &nic->eth_dev->data->dev_link);
- _rte_eth_dev_callback_process(nic->eth_dev,
- RTE_ETH_EVENT_INTR_LSC);
+ if (dev->data->dev_conf.intr_conf.lsc)
+ nicvf_set_eth_link_status(nic, &dev->data->dev_link);
+ _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC);
}
rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
- nicvf_interrupt, nic);
+ nicvf_interrupt, dev);
+}
+
+static void __rte_unused
+nicvf_vf_interrupt(void *arg)
+{
+ struct nicvf *nic = arg;
+
+ nicvf_reg_poll_interrupts(nic);
+
+ rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
+ nicvf_vf_interrupt, nic);
}
static int
-nicvf_periodic_alarm_start(struct nicvf *nic)
+nicvf_periodic_alarm_start(void (fn)(void *), void *arg)
{
- return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
- nicvf_interrupt, nic);
+ return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000, fn, arg);
}
static int
-nicvf_periodic_alarm_stop(struct nicvf *nic)
+nicvf_periodic_alarm_stop(void (fn)(void *), void *arg)
{
- return rte_eal_alarm_cancel(nicvf_interrupt, nic);
+ return rte_eal_alarm_cancel(fn, arg);
}
/*
@@ -1519,12 +1528,10 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
static void
nicvf_dev_close(struct rte_eth_dev *dev)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
-
PMD_INIT_FUNC_TRACE();
nicvf_dev_stop(dev);
- nicvf_periodic_alarm_stop(nic);
+ nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
}
static int
@@ -1675,7 +1682,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
nicvf_disable_all_interrupts(nic);
- ret = nicvf_periodic_alarm_start(nic);
+ ret = nicvf_periodic_alarm_start(nicvf_interrupt, eth_dev);
if (ret) {
PMD_INIT_LOG(ERR, "Failed to start period alarm");
goto fail;
@@ -1736,7 +1743,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
malloc_fail:
rte_free(eth_dev->data->mac_addrs);
alarm_fail:
- nicvf_periodic_alarm_stop(nic);
+ nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
fail:
return ret;
}
--
1.9.1
[dpdk-dev] [PATCH v2 07/15] net/thunderx: remove problematic private_data->eth_dev link
From: Kamil Rytarowski
In case of the multiprocess mode a shared nicvf struct between processes
cannot point with the eth_dev pointer to master device, therefore remove it
allong with references to it refactoring the code where needed.
This change fixes multiprocess issues detected in stats.
Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 69 ++---
drivers/net/thunderx/nicvf_rxtx.c | 3 +-
drivers/net/thunderx/nicvf_struct.h | 1 -
3 files changed, 36 insertions(+), 37 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 72e6667..c8b42f7 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -497,14 +497,14 @@ nicvf_dev_rss_hash_update(struct rte_eth_dev *dev,
}
static int
-nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
- uint32_t desc_cnt)
+nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ struct nicvf_rxq *rxq, uint16_t qidx, uint32_t desc_cnt)
{
const struct rte_memzone *rz;
uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "cq_ring", qidx, ring_size,
- NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+ NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
return -ENOMEM;
@@ -520,13 +520,13 @@ nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq
*rxq, uint16_t qidx,
}
static int
-nicvf_qset_sq_alloc(struct nicvf *nic, struct nicvf_txq *sq, uint16_t qidx,
- uint32_t desc_cnt)
+nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ struct nicvf_txq *sq, uint16_t qidx, uint32_t desc_cnt)
{
const struct rte_memzone *rz;
uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "sq", qidx, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
@@ -543,7 +543,8 @@ nicvf_qset_sq_alloc(struct nicvf *nic, struct nicvf_txq
*sq, uint16_t qidx,
}
static int
-nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
+nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint32_t desc_cnt, uint32_t buffsz)
{
struct nicvf_rbdr *rbdr;
const struct rte_memzone *rz;
@@ -558,7 +559,7 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt,
uint32_t buffsz)
}
ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "rbdr", 0, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
@@ -583,14 +584,15 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t
desc_cnt, uint32_t buffsz)
}
static void
-nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic
__rte_unused,
+ nicvf_phys_addr_t phy)
{
uint16_t qidx;
void *obj;
struct nicvf_rxq *rxq;
- for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
- rxq = nic->eth_dev->data->rx_queues[qidx];
+ for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+ rxq = dev->data->rx_queues[qidx];
if (rxq->precharge_cnt) {
obj = (void *)nicvf_mbuff_phy2virt(phy,
rxq->mbuf_phys_off);
@@ -602,7 +604,7 @@ nicvf_rbdr_release_mbuf(struct nicvf *nic,
nicvf_phys_addr_t phy)
}
static inline void
-nicvf_rbdr_release_mbufs(struct nicvf *nic)
+nicvf_rbdr_release_mbufs(struct rte_eth_dev *dev, struct nicvf *nic)
{
uint32_t qlen_mask, head;
struct rbdr_entry_t *entry;
@@ -612,7 +614,7 @@ nicvf_rbdr_release_mbufs(struct nicvf *nic)
head = rbdr->head;
while (head != rbdr->t
[dpdk-dev] [PATCH v2 08/15] net/thunderx: add helper utils for secondary qset support
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.h | 39 +
1 file changed, 39 insertions(+)
diff --git a/drivers/net/thunderx/nicvf_ethdev.h
b/drivers/net/thunderx/nicvf_ethdev.h
index 34447e0..e162364 100644
--- a/drivers/net/thunderx/nicvf_ethdev.h
+++ b/drivers/net/thunderx/nicvf_ethdev.h
@@ -87,6 +87,17 @@ nicvf_mbuff_meta_length(struct rte_mbuf *mbuf)
return (uint16_t)((uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf);
}
+static inline uint16_t
+nicvf_netdev_qidx(struct nicvf *nic, uint8_t local_qidx)
+{
+ uint16_t global_qidx = local_qidx;
+
+ if (nic->sqs_mode)
+ global_qidx += ((nic->sqs_id + 1) * MAX_CMP_QUEUES_PER_QS);
+
+ return global_qidx;
+}
+
/*
* Simple phy2virt functions assuming mbufs are in a single huge page
* V = P + offset
@@ -104,4 +115,32 @@ nicvf_mbuff_virt2phy(uintptr_t virt, uint64_t
mbuf_phys_off)
return (phys_addr_t)(virt - mbuf_phys_off);
}
+static inline void
+nicvf_tx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *tx_start,
+ uint16_t *tx_end)
+{
+ uint16_t tmp;
+
+ *tx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+ MAX_SND_QUEUES_PER_QS);
+ tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+MAX_SND_QUEUES_PER_QS) - 1;
+ *tx_end = dev->data->nb_tx_queues ?
+ RTE_MIN(tmp, dev->data->nb_tx_queues - 1) : 0;
+}
+
+static inline void
+nicvf_rx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *rx_start,
+ uint16_t *rx_end)
+{
+ uint16_t tmp;
+
+ *rx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+ MAX_RCV_QUEUES_PER_QS);
+ tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+MAX_RCV_QUEUES_PER_QS) - 1;
+ *rx_end = dev->data->nb_rx_queues ?
+ RTE_MIN(tmp, dev->data->nb_rx_queues - 1) : 0;
+}
+
#endif /* __THUNDERX_NICVF_ETHDEV_H__ */
--
1.9.1
[dpdk-dev] [PATCH v2 09/15] net/thunderx: add secondary qset support in dev stop/close
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 142 +++-
1 file changed, 107 insertions(+), 35 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index c8b42f7..7ca7d07 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -67,9 +67,13 @@
#include "nicvf_ethdev.h"
#include "nicvf_rxtx.h"
+#include "nicvf_svf.h"
#include "nicvf_logs.h"
static void nicvf_dev_stop(struct rte_eth_dev *dev);
+static void nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup);
+static void nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic,
+ bool cleanup);
static inline int
nicvf_atomic_write_link_status(struct rte_eth_dev *dev,
@@ -674,23 +678,29 @@ config_sq_error:
}
static inline int
-nicvf_stop_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
struct nicvf_txq *txq;
int ret;
- if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+ assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+ if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STOPPED)
return 0;
- ret = nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+ ret = nicvf_qset_sq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim sq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim sq VF%d %d %d",
+nic->vf_id, qidx, ret);
- txq = dev->data->tx_queues[qidx];
+ txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
nicvf_tx_queue_release_mbufs(txq);
nicvf_tx_queue_reset(txq);
- dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+ dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
return ret;
}
@@ -1002,30 +1012,34 @@ config_rq_error:
}
static inline int
-nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
struct nicvf_rxq *rxq;
int ret, other_error;
- if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+ if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STOPPED)
return 0;
ret = nicvf_qset_rq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim rq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim rq VF%d %d %d",
+nic->vf_id, qidx, ret);
other_error = ret;
- rxq = dev->data->rx_queues[qidx];
+ rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
nicvf_rx_queue_release_mbufs(dev, rxq);
nicvf_rx_queue_reset(rxq);
ret = nicvf_qset_cq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim cq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim cq VF%d %d %d",
+nic->vf_id, qidx, ret);
other_error |= ret;
- dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+ dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
return other_error;
}
@@ -1057,8 +1071,14 @@ static int
nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
{
int ret;
+ struct nicvf *nic = nicvf_pmd_priv(dev);
- ret = nicvf_stop_rx_queue(dev, qidx);
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+ ret = nicvf_vf_stop_rx_queue(dev, nic, qidx);
ret |= nicvf_configure_cpi(dev);
ret |= nicvf_configure_rss_reta(dev);
return ret;
@@ -1073,7 +1093,14 @@ nicvf_dev_tx_queue_start(struct rte_eth_dev *dev,
uint16_t qidx)
static int
nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
{
- return nicvf_stop_tx_queue(dev, qidx);
+ struct nicvf *nic = nicvf_pmd_priv(dev);
+
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+ re
[dpdk-dev] [PATCH v2 10/15] net/thunderx: add secondary qset support in device start
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 266 +++-
1 file changed, 172 insertions(+), 94 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 7ca7d07..f7d5188 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -653,27 +653,33 @@ nicvf_tx_queue_reset(struct nicvf_txq *txq)
}
static inline int
-nicvf_start_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
struct nicvf_txq *txq;
int ret;
- if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+ assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+ if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STARTED)
return 0;
- txq = dev->data->tx_queues[qidx];
+ txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
txq->pool = NULL;
- ret = nicvf_qset_sq_config(nicvf_pmd_priv(dev), qidx, txq);
+ ret = nicvf_qset_sq_config(nic, qidx, txq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure sq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure sq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_sq_error;
}
- dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+ dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STARTED;
return ret;
config_sq_error:
- nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+ nicvf_qset_sq_reclaim(nic, qidx);
return ret;
}
@@ -977,31 +983,37 @@ nicvf_rx_queue_reset(struct nicvf_rxq *rxq)
}
static inline int
-nicvf_start_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
struct nicvf_rxq *rxq;
int ret;
- if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+ assert(qidx < MAX_RCV_QUEUES_PER_QS);
+
+ if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STARTED)
return 0;
/* Update rbdr pointer to all rxq */
- rxq = dev->data->rx_queues[qidx];
+ rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
rxq->shared_rbdr = nic->rbdr;
ret = nicvf_qset_rq_config(nic, qidx, rxq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure rq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure rq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_rq_error;
}
ret = nicvf_qset_cq_config(nic, qidx, rxq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure cq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure cq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_cq_error;
}
- dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+ dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STARTED;
return 0;
config_cq_error:
@@ -1054,9 +1066,15 @@ nicvf_dev_rx_queue_release(void *rx_queue)
static int
nicvf_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
{
+ struct nicvf *nic = nicvf_pmd_priv(dev);
int ret;
- ret = nicvf_start_rx_queue(dev, qidx);
+ if (qidx >= MAX_RCV_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_RCV_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+ ret = nicvf_vf_start_rx_queue(dev, nic, qidx);
if (ret)
return ret;
@@ -1087,7 +1105,14 @@ nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev,
uint16_t qidx)
static int
nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
{
- return nicvf_start_tx_queue(dev, qidx);
+ struct nicvf *nic = nicvf_pmd_priv(dev);
+
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+ return nicvf_vf_start_tx_queue(dev, nic, qidx);
}
static int
@@ -1274,25 +1299,25 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
}
static int
-nicvf_dev_start(struct rte_eth_d
[dpdk-dev] [PATCH v2 11/15] net/thunderx: add secondary qset support in device configure
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 65 +
1 file changed, 59 insertions(+), 6 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index f7d5188..5f12b91 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -163,6 +163,7 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
{
struct nicvf *nic = nicvf_pmd_priv(dev);
uint32_t buffsz, frame_size = mtu + ETHER_HDR_LEN + ETHER_CRC_LEN;
+ size_t i;
PMD_INIT_FUNC_TRACE();
@@ -198,6 +199,10 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
/* Update max frame size */
dev->data->dev_conf.rxmode.max_rx_pkt_len = (uint32_t)frame_size;
nic->mtu = mtu;
+
+ for (i = 0; i < nic->sqs_count; i++)
+ nic->snicvf[i]->mtu = mtu;
+
return 0;
}
@@ -507,7 +512,8 @@ nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
const struct rte_memzone *rz;
uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
- rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "cq_ring",
+ nicvf_netdev_qidx(nic, qidx), ring_size,
NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
@@ -530,8 +536,9 @@ nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
const struct rte_memzone *rz;
uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
- rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
- NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "sq",
+ nicvf_netdev_qidx(nic, qidx), ring_size,
+ NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
return -ENOMEM;
@@ -563,8 +570,9 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
}
ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
- rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
- NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "rbdr",
+ nicvf_netdev_qidx(nic, 0), ring_size,
+ NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
return -ENOMEM;
@@ -1685,12 +1693,37 @@ nicvf_dev_close(struct rte_eth_dev *dev)
}
static int
+nicvf_request_sqs(struct nicvf *nic)
+{
+ size_t i;
+
+ assert_primary(nic);
+ assert(nic->sqs_count > 0);
+ assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+ /* Set no of Rx/Tx queues in each of the SQsets */
+ for (i = 0; i < nic->sqs_count; i++) {
+ if (nicvf_svf_empty())
+ rte_panic("Cannot assign sufficient number of "
+ "secondary queues to primary VF%" PRIu8 "\n",
+ nic->vf_id);
+
+ nic->snicvf[i] = nicvf_svf_pop();
+ nic->snicvf[i]->sqs_id = i;
+ }
+
+ return nicvf_mbox_request_sqs(nic);
+}
+
+static int
nicvf_dev_configure(struct rte_eth_dev *dev)
{
- struct rte_eth_conf *conf = &dev->data->dev_conf;
+ struct rte_eth_dev_data *data = dev->data;
+ struct rte_eth_conf *conf = &data->dev_conf;
struct rte_eth_rxmode *rxmode = &conf->rxmode;
struct rte_eth_txmode *txmode = &conf->txmode;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint8_t cqcount;
PMD_INIT_FUNC_TRACE();
@@ -1755,6 +1788,26 @@ nicvf_dev_configure(struct rte_eth_dev *dev)
return -EINVAL;
}
+ assert_primary(nic);
+ NICVF_STATIC_ASSERT(MAX_RCV_QUEUES_PER_QS == MAX_SND_QUEUES_PER_QS);
+ cqcount = RTE_MAX(data->nb_tx_queues, data->nb_rx_queues);
+ if (cqcount > MAX_RCV_QUEUES_PER_QS) {
+ nic->sqs_count = RTE_ALIGN_CEIL(cqcount, MAX_RCV_QUEUES_PER_QS);
+ nic->sqs_count = (nic->sqs_count / MAX_RCV_QUEUES_PER_QS) - 1;
+ } else {
+
[dpdk-dev] [PATCH v2 12/15] net/thunderx: add final bits for secondary queue support
From: Kamil Rytarowski
Summary:
- add secondary qset support in device stats
- add support for releasing mbufs from RBDR for >8 queues
- add support for releasing mbufs from RX queues for >8 queues
- support >8 queues in tx_queue_setup
- support >8 queues in rx_queue_setup
- support up to 96 queues per device (dev_info->max_rx_queues)
- add secondary qset support in rbdr_rte_mempool_get
- support >8 queues in multiprocess mode (do not reconfigure VFs)
- setup periodic alarm accordingly for type of VFs:
* primary VF - handle events on queues and link status
* secondary VF - handle events on queues
- initialize hardware capabilities in secondary qsets
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 200
1 file changed, 157 insertions(+), 43 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 5f12b91..9dde75d 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -118,7 +118,7 @@ nicvf_interrupt(void *arg)
nicvf_interrupt, dev);
}
-static void __rte_unused
+static void
nicvf_vf_interrupt(void *arg)
{
struct nicvf *nic = arg;
@@ -236,9 +236,15 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
struct nicvf_hw_tx_qstats tx_qstats;
struct nicvf_hw_stats port_stats;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint16_t rx_start, rx_end;
+ uint16_t tx_start, tx_end;
+ size_t i;
+
+ /* RX queue indices for the first VF */
+ nicvf_rx_range(dev, nic, &rx_start, &rx_end);
/* Reading per RX ring stats */
- for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+ for (qidx = rx_start; qidx <= rx_end; qidx++) {
if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
break;
@@ -247,8 +253,11 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
}
+ /* TX queue indices for the first VF */
+ nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
/* Reading per TX ring stats */
- for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
+ for (qidx = tx_start; qidx <= tx_end; qidx++) {
if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
break;
@@ -257,6 +266,40 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
}
+ for (i = 0; i < nic->sqs_count; i++) {
+ struct nicvf *snic = nic->snicvf[i];
+
+ if (snic == NULL)
+ break;
+
+ /* RX queue indices for a secondary VF */
+ nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+
+ /* Reading per RX ring stats */
+ for (qidx = rx_start; qidx <= rx_end; qidx++) {
+ if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+ break;
+
+ nicvf_hw_get_rx_qstats(snic, &rx_qstats,
+ qidx % MAX_RCV_QUEUES_PER_QS);
+ stats->q_ibytes[qidx] = rx_qstats.q_rx_bytes;
+ stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
+ }
+
+ /* TX queue indices for a secondary VF */
+ nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+ /* Reading per TX ring stats */
+ for (qidx = tx_start; qidx <= tx_end; qidx++) {
+ if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+ break;
+
+ nicvf_hw_get_tx_qstats(snic, &tx_qstats,
+ qidx % MAX_SND_QUEUES_PER_QS);
+ stats->q_obytes[qidx] = tx_qstats.q_tx_bytes;
+ stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
+ }
+ }
+
nicvf_hw_get_stats(nic, &port_stats);
stats->ibytes = port_stats.rx_bytes;
stats->ipackets = port_stats.rx_ucast_frames;
@@ -322,13 +365,36 @@ nicvf_dev_stats_reset(struct rte_eth_dev *dev)
int i;
uint16_t rxqs = 0, txqs = 0;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint16_t rx_start, rx_end;
+ uint16_t tx_start, tx_end;
- for (i = 0; i < dev->data->nb_rx_queues; i++)
+ /* Reset all primary nic counters */
+ nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+ for (i = rx_start; i <= rx_end; i++)
[dpdk-dev] [PATCH v2 13/15] net/thunderx: document secondary queue set support
From: Kamil Rytarowski Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- doc/guides/nics/thunderx.rst | 148 --- 1 file changed, 124 insertions(+), 24 deletions(-) diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index acc05a4..9763bb6 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -56,6 +56,7 @@ Features of the ThunderX PMD are: - VLAN stripping - SR-IOV VF - NUMA support +- Multi queue set support (up to 96 queues (12 queue sets)) per port Supported ThunderX SoCs --- @@ -207,13 +208,13 @@ This section provides instructions to configure SR-IOV with Linux OS. .. code-block:: console 0002:01:00.0 Ethernet controller: Cavium Networks Device a01e (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) - ... - Kernel driver in use: thunder-nic - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) + ... + Kernel driver in use: thunder-nic + ... .. note:: @@ -230,18 +231,18 @@ This section provides instructions to configure SR-IOV with Linux OS. .. code-block:: console 0002:01:00.1 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... 0002:01:00.2 Ethernet controller: Cavium Networks Device 0011 (rev 01) - ... - Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) - ... - Kernel driver in use: thunder-nicvf - ... + ... + Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) + ... + Kernel driver in use: thunder-nicvf + ... .. note:: @@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS. #. Refer to section :ref:`Running testpmd ` for instruction how to launch ``testpmd`` application. +Multiple Queue Set per DPDK port configuration +~~ + +There are two types of VFs: + +- Primary VF +- Secondary VF + +Each port consists of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. +When a given port is configured to use more than 8 queues, it requires one (or more) secondary VF. +Each secondary VF adds 8 additional queues to the queue set. + +During PMD driver initialization, the primary VF's are enumerated by checking the +specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set). +They are at the beginning of VF list (the remain ones are secondary VF's). + +The primary VFs are used as master queue sets. Secondary VFs provide +additional queue sets for primary ones. If a port is configured for more then +8 queues than it will request for additional queues from secondary VFs. + +Secondary VFs cannot be shared between primary VFs. + +Primary VFs are present on the beginning of the 'Network devices using kernel +driver' list, secondary VFs are on the remaining on the remaining part of the list. + + .. note:: + + The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. + We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. + + .. note:: + + Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to + this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. + + +Example device binding +~~ + +If a system has three interfaces, a total of 18 VF devices will be created +on a non-NUMA machine. + + .. note:: + + NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port. + + .. code-block:: console + + # tools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + + + + Network devices using kernel driver + === + :01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + :01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio
[dpdk-dev] [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains
It's possible to have the same numbers for bus, device id and function,
therefore we need to differentiate on domain.
This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
lib/librte_ether/rte_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 382c959..01d5fb0 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t
size,
{
int ret;
- ret = snprintf(name, size, "%d:%d.%d",
+ ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
pci_dev->addr.bus, pci_dev->addr.devid,
pci_dev->addr.function);
if (ret < 0)
--
1.9.1
[dpdk-dev] [PATCH v2 15/15] net/thunderx: Bump driver version to 2.0
Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- drivers/net/thunderx/nicvf_ethdev.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/thunderx/nicvf_ethdev.h b/drivers/net/thunderx/nicvf_ethdev.h index e162364..a74219f 100644 --- a/drivers/net/thunderx/nicvf_ethdev.h +++ b/drivers/net/thunderx/nicvf_ethdev.h @@ -35,7 +35,7 @@ #include -#define THUNDERX_NICVF_PMD_VERSION "1.0" +#define THUNDERX_NICVF_PMD_VERSION "2.0" #define THUNDERX_REG_BYTES 8 #define NICVF_INTR_POLL_INTERVAL_MS50 -- 1.9.1
[dpdk-dev] [PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver
ping W dniu 26.08.2016 o 18:53, Kamil Rytarowski pisze: > This series of patches adds support for secondary queue set in nicvf thunderx > driver > > There are two types of VFs: > - Primary VF > - Secondary VF > > Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 > Tx/Rx queues to a port. In case port is configured to use more than 8 queues, > then it requires one (or more) secondary VF. Each secondary VF adds additional > 8 queues to the queue set. > > During PMD driver initialization, the primary VF's are enumerated by checking > the > specific flag (see READY message). They are at the beginning of VF list (the > remain > ones are secondary VF's). > > The primary VFs are used as master queue sets. Secondary VFs provide > additional queue sets for primary ones. If a port is configured for more then > 8 queues then it will request for additional queues from secondary VFs. > > Secondary VFs cannot be shared between primary VFs. > > Primary VFs are present on the tail of the 'Network devices using kernel > driver' list, secondary VFs are on the remaining tail of the list. > > The VNIC driver in the multiqueue setup works differently than other drivers > like `ixgbe`. We need to bind separately each specific queue set device with > the ``tools/dpdk-devbind.py`` utility. > > Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. > VFs that try to attach with an id below or equal to this boundary are > considered primary VFs. VFs that try to attach with an id above this boundary > are considered secondary VFs. > > This patchset also contains other cleanups and improvements like fixing > erroneous checksum calculation and preparing the thunderx driver for the multi > queue set feature support. > > > These changes base on the following pending patches: > > [dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver > http://dpdk.org/dev/patchwork/patch/14963/ > > [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag > http://dpdk.org/dev/patchwork/patch/14964/ > > [dpdk-dev,3/3] net/thunderx: add 81xx SoC support > http://dpdk.org/dev/patchwork/patch/14965/ > > Kamil Rytarowski (13): >net/thunderx: cleanup the driver before adding new features >net/thunderx: correct transmit checksum handling >net/thunderx/base: add family of functions to store qsets >net/thunderx/base: add secondary queue set support >net/thunderx: add family of functions to store DPDK qsets >net/thunderx: add secondary queue set in interrupt functions >net/thunderx: fix multiprocess support in stats >net/thunderx: add helper utils for secondary qset support >net/thunderx: add secondary qset support in dev stop/close >net/thunderx: add secondary qset support in device start >net/thunderx: add secondary qset support in device configure >net/thunderx: add final bits for secondary queue support >net/thunderx: document secondary queue set support > > doc/guides/nics/thunderx.rst | 114 - > drivers/net/thunderx/Makefile | 2 + > drivers/net/thunderx/base/nicvf_bsvf.c| 72 +++ > drivers/net/thunderx/base/nicvf_bsvf.h| 76 +++ > drivers/net/thunderx/base/nicvf_hw.c | 10 +- > drivers/net/thunderx/base/nicvf_hw.h | 6 +- > drivers/net/thunderx/base/nicvf_hw_defs.h | 1 + > drivers/net/thunderx/base/nicvf_mbox.c| 34 +- > drivers/net/thunderx/base/nicvf_mbox.h| 21 +- > drivers/net/thunderx/nicvf_ethdev.c | 753 > +- > drivers/net/thunderx/nicvf_ethdev.h | 39 ++ > drivers/net/thunderx/nicvf_rxtx.c | 14 +- > drivers/net/thunderx/nicvf_struct.h | 6 +- > drivers/net/thunderx/nicvf_svf.c | 78 > drivers/net/thunderx/nicvf_svf.h | 66 +++ > 15 files changed, 1046 insertions(+), 246 deletions(-) > create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c > create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h > create mode 100644 drivers/net/thunderx/nicvf_svf.c > create mode 100644 drivers/net/thunderx/nicvf_svf.h >
[dpdk-dev] [PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver
ping W dniu 12.09.2016 o 12:59, Kamil Rytarowski pisze: > ping > > > W dniu 26.08.2016 o 18:53, Kamil Rytarowski pisze: >> This series of patches adds support for secondary queue set in nicvf >> thunderx >> driver >> >> There are two types of VFs: >> - Primary VF >> - Secondary VF >> >> Each port consist of a primary VF and n secondary VF(s). Each VF >> provides 8 >> Tx/Rx queues to a port. In case port is configured to use more than 8 >> queues, >> then it requires one (or more) secondary VF. Each secondary VF adds >> additional >> 8 queues to the queue set. >> >> During PMD driver initialization, the primary VF's are enumerated by >> checking the >> specific flag (see READY message). They are at the beginning of VF >> list (the remain >> ones are secondary VF's). >> >> The primary VFs are used as master queue sets. Secondary VFs provide >> additional queue sets for primary ones. If a port is configured for >> more then >> 8 queues then it will request for additional queues from secondary VFs. >> >> Secondary VFs cannot be shared between primary VFs. >> >> Primary VFs are present on the tail of the 'Network devices using kernel >> driver' list, secondary VFs are on the remaining tail of the list. >> >> The VNIC driver in the multiqueue setup works differently than other >> drivers >> like `ixgbe`. We need to bind separately each specific queue set >> device with >> the ``tools/dpdk-devbind.py`` utility. >> >> Depending on the hardware used, the kernel driver sets a threshold >> ``vf_id``. >> VFs that try to attach with an id below or equal to this boundary are >> considered primary VFs. VFs that try to attach with an id above this >> boundary >> are considered secondary VFs. >> >> This patchset also contains other cleanups and improvements like fixing >> erroneous checksum calculation and preparing the thunderx driver for >> the multi >> queue set feature support. >> >> >> These changes base on the following pending patches: >> >> [dpdk-dev,1/3] net/thunderx: remove generic passx references from the >> driver >> http://dpdk.org/dev/patchwork/patch/14963/ >> >> [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag >> http://dpdk.org/dev/patchwork/patch/14964/ >> >> [dpdk-dev,3/3] net/thunderx: add 81xx SoC support >> http://dpdk.org/dev/patchwork/patch/14965/ >> >> Kamil Rytarowski (13): >>net/thunderx: cleanup the driver before adding new features >>net/thunderx: correct transmit checksum handling >>net/thunderx/base: add family of functions to store qsets >>net/thunderx/base: add secondary queue set support >>net/thunderx: add family of functions to store DPDK qsets >>net/thunderx: add secondary queue set in interrupt functions >>net/thunderx: fix multiprocess support in stats >>net/thunderx: add helper utils for secondary qset support >>net/thunderx: add secondary qset support in dev stop/close >>net/thunderx: add secondary qset support in device start >>net/thunderx: add secondary qset support in device configure >>net/thunderx: add final bits for secondary queue support >>net/thunderx: document secondary queue set support >> >> doc/guides/nics/thunderx.rst | 114 - >> drivers/net/thunderx/Makefile | 2 + >> drivers/net/thunderx/base/nicvf_bsvf.c| 72 +++ >> drivers/net/thunderx/base/nicvf_bsvf.h| 76 +++ >> drivers/net/thunderx/base/nicvf_hw.c | 10 +- >> drivers/net/thunderx/base/nicvf_hw.h | 6 +- >> drivers/net/thunderx/base/nicvf_hw_defs.h | 1 + >> drivers/net/thunderx/base/nicvf_mbox.c| 34 +- >> drivers/net/thunderx/base/nicvf_mbox.h| 21 +- >> drivers/net/thunderx/nicvf_ethdev.c | 753 >> +- >> drivers/net/thunderx/nicvf_ethdev.h | 39 ++ >> drivers/net/thunderx/nicvf_rxtx.c | 14 +- >> drivers/net/thunderx/nicvf_struct.h | 6 +- >> drivers/net/thunderx/nicvf_svf.c | 78 >> drivers/net/thunderx/nicvf_svf.h | 66 +++ >> 15 files changed, 1046 insertions(+), 246 deletions(-) >> create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c >> create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h >> create mode 100644 drivers/net/thunderx/nicvf_svf.c >> create mode 100644 drivers/net/thunderx/nicvf_svf.h >> >
[dpdk-dev] [PATCH] tools: Add support for handling built-in kernel modules
Currently dpdk_nic_bind.py detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files.
Signed-off-by: Kamil Rytarowski
---
tools/dpdk_nic_bind.py | 22 +-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
index 8523f82..2ddc14a 100755
--- a/tools/dpdk_nic_bind.py
+++ b/tools/dpdk_nic_bind.py
@@ -33,7 +33,7 @@
#
import sys, os, getopt, subprocess
-from os.path import exists, abspath, dirname, basename
+from os.path import exists, abspath, dirname, basename, join
# The PCI device class for ETHERNET devices
@@ -173,6 +173,26 @@ def check_modules():
elif line.replace("_", "-").startswith(mod["Name"]):
mod["Found"] = True
+try:
+# Get list of syfs modules, some of them might be builtin and merge
with mods
+sysfs_path = '/sys/module/'
+
+# Get the list of directories in sysfs_path
+sysfs_mods = [os.path.join(sysfs_path,o) for o in
os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
+
+# Extract the last element of '/sys/module/abc' in the array
+sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
+
+# special case for vfio_pci (module is named vfio-pci,
+# but its .ko is named vfio_pci)
+sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
sysfs_mods)
+
+for mod in mods:
+if mod["Found"] == False and (mod["Name"] in sysfs_mods):
+mod["Found"] = True
+except:
+pass
+
# check if we have at least one loaded module
if True not in [mod["Found"] for mod in mods] and b_flag is not None:
if b_flag in dpdk_drivers:
--
2.5.0
[dpdk-dev] [PATCH v2 1/2] tools: Add support for handling built-in kernel modules
Currently dpdk_nic_bind.py detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files.
Signed-off-by: Kamil Rytarowski
---
tools/dpdk_nic_bind.py | 22 +-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
index f02454e..cb1ac80 100755
--- a/tools/dpdk_nic_bind.py
+++ b/tools/dpdk_nic_bind.py
@@ -33,7 +33,7 @@
#
import sys, os, getopt, subprocess
-from os.path import exists, abspath, dirname, basename
+from os.path import exists, abspath, dirname, basename, join
# The PCI device class for ETHERNET devices
@@ -173,6 +173,26 @@ def check_modules():
elif line.replace("_", "-").startswith(mod["Name"]):
mod["Found"] = True
+try:
+# Get list of syfs modules, some of them might be builtin and merge
with mods
+sysfs_path = '/sys/module/'
+
+# Get the list of directories in sysfs_path
+sysfs_mods = [os.path.join(sysfs_path,o) for o in
os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
+
+# Extract the last element of '/sys/module/abc' in the array
+sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
+
+# special case for vfio_pci (module is named vfio-pci,
+# but its .ko is named vfio_pci)
+sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
sysfs_mods)
+
+for mod in mods:
+if mod["Found"] == False and (mod["Name"] in sysfs_mods):
+mod["Found"] = True
+except:
+pass
+
# check if we have at least one loaded module
if True not in [mod["Found"] for mod in mods] and b_flag is not None:
if b_flag in dpdk_drivers:
--
2.5.0
[dpdk-dev] [PATCH v2 2/2] eal/linux: Add support for handling built-in kernel modules
Currently rte_eal_check_module() detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files
Signed-off-by: Kamil Rytarowski
---
lib/librte_eal/linuxapp/eal/eal.c | 21 -
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c
b/lib/librte_eal/linuxapp/eal/eal.c
index 635ec36..6cab906 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -52,6 +52,8 @@
#if defined(RTE_ARCH_X86_64) || defined(RTE_ARCH_I686)
#include
#endif
+#include
+#include
#include
#include
@@ -902,7 +904,10 @@ int
rte_eal_check_module(const char *module_name)
{
char mod_name[30]; /* Any module names can be longer than 30 bytes? */
+ char sysfs_mod_name[PATH_MAX];
+ struct stat st;
int ret = 0;
+ int rv;
int n;
if (NULL == module_name)
@@ -918,9 +923,23 @@ rte_eal_check_module(const char *module_name)
n = fscanf(fd, "%29s %*[^\n]", mod_name);
if ((n == 1) && !strcmp(mod_name, module_name)) {
ret = 1;
- break;
+ goto finish;
}
}
+ RTE_LOG(DEBUG, EAL, "Module %s not found in /proc/modules",
+ module_name);
+
+ /* A module might be builtin, try sysfs */
+ snprintf(sysfs_mod_name, PATH_MAX, "/sys/module/%s", module_name);
+ if ((rv = stat(sysfs_mod_name, &st)) == 0) {
+ ret = 1;
+ goto finish;
+ }
+
+ RTE_LOG(DEBUG, EAL, "Open %s failed! error %i (%s)\n",
+ sysfs_mod_name, errno, strerror(errno));
+
+finish:
fclose(fd);
return ret;
--
2.5.0
[dpdk-dev] [PATCH v3 1/2] tools: Add support for handling built-in kernel modules
Currently dpdk_nic_bind.py detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files.
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
---
tools/dpdk_nic_bind.py | 29 ++---
1 file changed, 18 insertions(+), 11 deletions(-)
diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
index f02454e..0318598 100755
--- a/tools/dpdk_nic_bind.py
+++ b/tools/dpdk_nic_bind.py
@@ -33,7 +33,7 @@
#
import sys, os, getopt, subprocess
-from os.path import exists, abspath, dirname, basename
+from os.path import exists, abspath, dirname, basename, join
# The PCI device class for ETHERNET devices
@@ -156,22 +156,29 @@ def check_modules():
'''Checks that igb_uio is loaded'''
global dpdk_drivers
-fd = file("/proc/modules")
-loaded_mods = fd.readlines()
-fd.close()
-
# list of supported modules
mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
# first check if module is loaded
-for line in loaded_mods:
+try:
+# Get list of syfs modules, some of them might be builtin and merge
with mods
+sysfs_path = '/sys/module/'
+
+# Get the list of directories in sysfs_path
+sysfs_mods = [os.path.join(sysfs_path,o) for o in
os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
+
+# Extract the last element of '/sys/module/abc' in the array
+sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
+
+# special case for vfio_pci (module is named vfio-pci,
+# but its .ko is named vfio_pci)
+sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
sysfs_mods)
+
for mod in mods:
-if line.startswith(mod["Name"]):
-mod["Found"] = True
-# special case for vfio_pci (module is named vfio-pci,
-# but its .ko is named vfio_pci)
-elif line.replace("_", "-").startswith(mod["Name"]):
+if mod["Found"] == False and (mod["Name"] in sysfs_mods):
mod["Found"] = True
+except:
+pass
# check if we have at least one loaded module
if True not in [mod["Found"] for mod in mods] and b_flag is not None:
--
2.5.0
[dpdk-dev] [PATCH v3 2/2] eal/linux: Add support for handling built-in kernel modules
Currently rte_eal_check_module() detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
---
lib/librte_eal/linuxapp/eal/eal.c | 32 +---
1 file changed, 17 insertions(+), 15 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c
b/lib/librte_eal/linuxapp/eal/eal.c
index 635ec36..539188f 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -52,6 +52,8 @@
#if defined(RTE_ARCH_X86_64) || defined(RTE_ARCH_I686)
#include
#endif
+#include
+#include
#include
#include
@@ -901,27 +903,27 @@ int rte_eal_has_hugepages(void)
int
rte_eal_check_module(const char *module_name)
{
- char mod_name[30]; /* Any module names can be longer than 30 bytes? */
- int ret = 0;
- int n;
+ char sysfs_mod_name[PATH_MAX];
+ struct stat st;
if (NULL == module_name)
return -1;
- FILE *fd = fopen("/proc/modules", "r");
- if (NULL == fd) {
- RTE_LOG(ERR, EAL, "Open /proc/modules failed!"
- " error %i (%s)\n", errno, strerror(errno));
+ /* Check if there is sysfs mounted */
+ if (stat("/sys/module", &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "Open /sys/module failed: %s\n",
+ strerror(errno));
return -1;
}
- while (!feof(fd)) {
- n = fscanf(fd, "%29s %*[^\n]", mod_name);
- if ((n == 1) && !strcmp(mod_name, module_name)) {
- ret = 1;
- break;
- }
+
+ /* A module might be built-in, therefore try sysfs */
+ snprintf(sysfs_mod_name, PATH_MAX, "/sys/module/%s", module_name);
+ if (stat(sysfs_mod_name, &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "Open %s failed! error %i (%s)\n",
+ sysfs_mod_name, errno, strerror(errno));
+ return 0;
}
- fclose(fd);
- return ret;
+ /* Module has been found */
+ return 1;
}
--
2.5.0
[dpdk-dev] [PATCH v3 2/2] eal/linux: Add support for handling built-in kernel modules
W dniu 08.12.2015 o 08:25, Panu Matilainen pisze:
> On 12/07/2015 10:55 PM, Stephen Hemminger wrote:
>> On Mon, 7 Dec 2015 19:36:05 +0100
>> Kamil Rytarowski wrote:
>>
>>> +/* Check if there is sysfs mounted */
>>> +if (stat("/sys/module", &st) != 0) {
>>> +RTE_LOG(DEBUG, EAL, "Open /sys/module failed: %s\n",
>>> +strerror(errno));
>>> return -1;
>>> }
>>
>> This check is useless.
>> If /sys/module does not exist then /sys/module/XXX won't exist either.
>
> Yes, but non-mounted sysfs is an error whereas /sys/module/XXX is
> merely an existence test, and the current sole caller in
> pci_vfio_enable() even bothers checking for the difference. So its
> perhaps a bit academic but its not incorrect.
>
> At any rate, the debug messages are incorrect/misleading. It's
> certainly not trying to *open* these directories so it should not
> claim to do so.
>
Yes, this check is to determine whether there is sysfs mounted. It's
different than checking if there is a module loaded.
This seems academical, but it retains the original behavior.
I will try to improve the logging.
> - Panu -
>
>
>
[dpdk-dev] [PATCH v4 1/2] tools: Add support for handling built-in kernel modules
Currently dpdk_nic_bind.py detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files.
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
Signed-off-by: David Marchand
---
tools/dpdk_nic_bind.py | 27 +--
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
index f02454e..e161062 100755
--- a/tools/dpdk_nic_bind.py
+++ b/tools/dpdk_nic_bind.py
@@ -156,22 +156,29 @@ def check_modules():
'''Checks that igb_uio is loaded'''
global dpdk_drivers
-fd = file("/proc/modules")
-loaded_mods = fd.readlines()
-fd.close()
-
# list of supported modules
mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
# first check if module is loaded
-for line in loaded_mods:
+try:
+# Get list of syfs modules, some of them might be builtin and merge
with mods
+sysfs_path = '/sys/module/'
+
+# Get the list of directories in sysfs_path
+sysfs_mods = [os.path.join(sysfs_path,o) for o in
os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
+
+# Extract the last element of '/sys/module/abc' in the array
+sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
+
+# special case for vfio_pci (module is named vfio-pci,
+# but its .ko is named vfio_pci)
+sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
sysfs_mods)
+
for mod in mods:
-if line.startswith(mod["Name"]):
-mod["Found"] = True
-# special case for vfio_pci (module is named vfio-pci,
-# but its .ko is named vfio_pci)
-elif line.replace("_", "-").startswith(mod["Name"]):
+if mod["Found"] == False and (mod["Name"] in sysfs_mods):
mod["Found"] = True
+except:
+pass
# check if we have at least one loaded module
if True not in [mod["Found"] for mod in mods] and b_flag is not None:
--
1.9.1
[dpdk-dev] [PATCH v4 2/2] eal/linux: Add support for handling built-in kernel modules
Currently rte_eal_check_module() detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
Signed-off-by: David Marchand
---
lib/librte_eal/linuxapp/eal/eal.c | 34 --
1 file changed, 20 insertions(+), 14 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c
b/lib/librte_eal/linuxapp/eal/eal.c
index 635ec36..92482a0 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -901,27 +901,33 @@ int rte_eal_has_hugepages(void)
int
rte_eal_check_module(const char *module_name)
{
- char mod_name[30]; /* Any module names can be longer than 30 bytes? */
- int ret = 0;
+ char sysfs_mod_name[PATH_MAX];
+ struct stat st;
int n;
if (NULL == module_name)
return -1;
- FILE *fd = fopen("/proc/modules", "r");
- if (NULL == fd) {
- RTE_LOG(ERR, EAL, "Open /proc/modules failed!"
- " error %i (%s)\n", errno, strerror(errno));
+ /* Check if there is sysfs mounted */
+ if (stat("/sys/module", &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "sysfs is not mounted! error %i (%s)\n",
+ errno, strerror(errno));
return -1;
}
- while (!feof(fd)) {
- n = fscanf(fd, "%29s %*[^\n]", mod_name);
- if ((n == 1) && !strcmp(mod_name, module_name)) {
- ret = 1;
- break;
- }
+
+ /* A module might be built-in, therefore try sysfs */
+ n = snprintf(sysfs_mod_name, PATH_MAX, "/sys/module/%s", module_name);
+ if (n < 0 || n > PATH_MAX) {
+ RTE_LOG(DEBUG, EAL, "Could not format module path\n");
+ return -1;
}
- fclose(fd);
- return ret;
+ if (stat(sysfs_mod_name, &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "Open %s failed! error %i (%s)\n",
+ sysfs_mod_name, errno, strerror(errno));
+ return 0;
+ }
+
+ /* Module has been found */
+ return 1;
}
--
1.9.1
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
Currently dpdk_nic_bind.py detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files.
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
Signed-off-by: David Marchand
---
tools/dpdk_nic_bind.py | 27 +--
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
index f02454e..e161062 100755
--- a/tools/dpdk_nic_bind.py
+++ b/tools/dpdk_nic_bind.py
@@ -156,22 +156,29 @@ def check_modules():
'''Checks that igb_uio is loaded'''
global dpdk_drivers
-fd = file("/proc/modules")
-loaded_mods = fd.readlines()
-fd.close()
-
# list of supported modules
mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
# first check if module is loaded
-for line in loaded_mods:
+try:
+# Get list of syfs modules, some of them might be builtin and merge
with mods
+sysfs_path = '/sys/module/'
+
+# Get the list of directories in sysfs_path
+sysfs_mods = [os.path.join(sysfs_path,o) for o in
os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
+
+# Extract the last element of '/sys/module/abc' in the array
+sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
+
+# special case for vfio_pci (module is named vfio-pci,
+# but its .ko is named vfio_pci)
+sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
sysfs_mods)
+
for mod in mods:
-if line.startswith(mod["Name"]):
-mod["Found"] = True
-# special case for vfio_pci (module is named vfio-pci,
-# but its .ko is named vfio_pci)
-elif line.replace("_", "-").startswith(mod["Name"]):
+if mod["Found"] == False and (mod["Name"] in sysfs_mods):
mod["Found"] = True
+except:
+pass
# check if we have at least one loaded module
if True not in [mod["Found"] for mod in mods] and b_flag is not None:
--
2.6.3
[dpdk-dev] [PATCH v5 2/2] eal/linux: Add support for handling built-in kernel modules
Currently rte_eal_check_module() detects Linux kernel modules via reading
/proc/modules. Built-in ones aren't listed there and therefore they are not
being found by the script.
Add support for checking built-in modules with parsing the sysfs files
This commit obsoletes the /proc/modules parsing approach.
Signed-off-by: Kamil Rytarowski
Signed-off-by: David Marchand
---
lib/librte_eal/linuxapp/eal/eal.c | 34 --
1 file changed, 20 insertions(+), 14 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c
b/lib/librte_eal/linuxapp/eal/eal.c
index 635ec36..21a4a32 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -901,27 +901,33 @@ int rte_eal_has_hugepages(void)
int
rte_eal_check_module(const char *module_name)
{
- char mod_name[30]; /* Any module names can be longer than 30 bytes? */
- int ret = 0;
+ char sysfs_mod_name[PATH_MAX];
+ struct stat st;
int n;
if (NULL == module_name)
return -1;
- FILE *fd = fopen("/proc/modules", "r");
- if (NULL == fd) {
- RTE_LOG(ERR, EAL, "Open /proc/modules failed!"
- " error %i (%s)\n", errno, strerror(errno));
+ /* Check if there is sysfs mounted */
+ if (stat("/sys/module", &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "sysfs is not mounted! error %i (%s)\n",
+ errno, strerror(errno));
return -1;
}
- while (!feof(fd)) {
- n = fscanf(fd, "%29s %*[^\n]", mod_name);
- if ((n == 1) && !strcmp(mod_name, module_name)) {
- ret = 1;
- break;
- }
+
+ /* A module might be built-in, therefore try sysfs */
+ n = snprintf(sysfs_mod_name, PATH_MAX, "/sys/module/%s", module_name);
+ if (n < 0 || n > PATH_MAX) {
+ RTE_LOG(DEBUG, EAL, "Could not format module path\n");
+ return -1;
}
- fclose(fd);
- return ret;
+ if (stat(sysfs_mod_name, &st) != 0) {
+ RTE_LOG(DEBUG, EAL, "Module %s not found! error %i (%s)\n",
+ sysfs_mod_name, errno, strerror(errno));
+ return 0;
+ }
+
+ /* Module has been found */
+ return 1;
}
--
2.6.3
[dpdk-dev] [PATCH v4 2/2] eal/linux: Add support for handling built-in kernel modules
W dniu 09.12.2015 o 08:30, Panu Matilainen pisze:
> On 12/08/2015 05:33 PM, Kamil Rytarowski wrote:
>> Currently rte_eal_check_module() detects Linux kernel modules via
>> reading
>> /proc/modules. Built-in ones aren't listed there and therefore they
>> are not
>> being found by the script.
>>
>> Add support for checking built-in modules with parsing the sysfs files
>>
>> This commit obsoletes the /proc/modules parsing approach.
>>
>> Signed-off-by: Kamil Rytarowski
>> Signed-off-by: David Marchand
>> ---
>> lib/librte_eal/linuxapp/eal/eal.c | 34
>> --
>> 1 file changed, 20 insertions(+), 14 deletions(-)
>>
>> diff --git a/lib/librte_eal/linuxapp/eal/eal.c
>> b/lib/librte_eal/linuxapp/eal/eal.c
>> index 635ec36..92482a0 100644
>> --- a/lib/librte_eal/linuxapp/eal/eal.c
>> +++ b/lib/librte_eal/linuxapp/eal/eal.c
>> @@ -901,27 +901,33 @@ int rte_eal_has_hugepages(void)
>> int
>> rte_eal_check_module(const char *module_name)
>> {
>> -char mod_name[30]; /* Any module names can be longer than 30
>> bytes? */
>> -int ret = 0;
>> +char sysfs_mod_name[PATH_MAX];
>> +struct stat st;
>> int n;
>>
>> if (NULL == module_name)
>> return -1;
>>
>> -FILE *fd = fopen("/proc/modules", "r");
>> -if (NULL == fd) {
>> -RTE_LOG(ERR, EAL, "Open /proc/modules failed!"
>> -" error %i (%s)\n", errno, strerror(errno));
>> +/* Check if there is sysfs mounted */
>> +if (stat("/sys/module", &st) != 0) {
>> +RTE_LOG(DEBUG, EAL, "sysfs is not mounted! error %i (%s)\n",
>> +errno, strerror(errno));
>> return -1;
>> }
>> -while (!feof(fd)) {
>> -n = fscanf(fd, "%29s %*[^\n]", mod_name);
>> -if ((n == 1) && !strcmp(mod_name, module_name)) {
>> -ret = 1;
>> -break;
>> -}
>> +
>> +/* A module might be built-in, therefore try sysfs */
>> +n = snprintf(sysfs_mod_name, PATH_MAX, "/sys/module/%s",
>> module_name);
>> +if (n < 0 || n > PATH_MAX) {
>> +RTE_LOG(DEBUG, EAL, "Could not format module path\n");
>> +return -1;
>> }
>> -fclose(fd);
>>
>> -return ret;
>> +if (stat(sysfs_mod_name, &st) != 0) {
>> +RTE_LOG(DEBUG, EAL, "Open %s failed! error %i (%s)\n",
>> +sysfs_mod_name, errno, strerror(errno));
>> +return 0;
>> +}
>
> Like with /sys/module, its not trying to *open* sysfs_mod_name
> directory either so it shouldn't claim to do so.
>
> I did use plural on purpose when I said "the debug messages are
> incorrect/misleading. It's certainly not trying to *open* these
> directories so it should not claim to do so" in my previous mail :)
>
> - Panu -
>
Should be good now!
Thank you.
[dpdk-dev] [PATCH v5 2/2] eal/linux: Add support for handling built-in kernel modules
W dniu 09.12.2015 o 17:38, Stephen Hemminger pisze:
> On Wed, 9 Dec 2015 14:19:58 +0100
> Kamil Rytarowski wrote:
>
>> +/* Check if there is sysfs mounted */
>> +if (stat("/sys/module", &st) != 0) {
>> +RTE_LOG(DEBUG, EAL, "sysfs is not mounted! error %i (%s)\n",
>> +errno, strerror(errno));
>> return -1;
>> }
> This check is redundant. Remove it.
> If the later "/sys/module/foo" would fail if /sys/module was not present.
This check isn't redundant. It checks different thing (sysfs mounted vs
module loaded) and formulates different error message.
Please see pci_vfio_enable():
module_vfio_type1 = rte_eal_check_module("vfio_iommu_type1");
/* return error directly */
if (module_vfio_type1 == -1) {
RTE_LOG(INFO, EAL, "Could not get loaded module
details!\n");
return -1;
}
/* return 0 if VFIO modules not loaded */
if (module_vfio_type1 == 0) {
RTE_LOG(INFO, EAL, "VFIO modules not all loaded, "
"skip VFIO support...\n");
return 0;
}
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
ping?
W dniu 09.12.2015 o 14:19, Kamil Rytarowski pisze:
> Currently dpdk_nic_bind.py detects Linux kernel modules via reading
> /proc/modules. Built-in ones aren't listed there and therefore they are not
> being found by the script.
>
> Add support for checking built-in modules with parsing the sysfs files.
>
> This commit obsoletes the /proc/modules parsing approach.
>
> Signed-off-by: Kamil Rytarowski
> Signed-off-by: David Marchand
> ---
> tools/dpdk_nic_bind.py | 27 +--
> 1 file changed, 17 insertions(+), 10 deletions(-)
>
> diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
> index f02454e..e161062 100755
> --- a/tools/dpdk_nic_bind.py
> +++ b/tools/dpdk_nic_bind.py
> @@ -156,22 +156,29 @@ def check_modules():
> '''Checks that igb_uio is loaded'''
> global dpdk_drivers
>
> -fd = file("/proc/modules")
> -loaded_mods = fd.readlines()
> -fd.close()
> -
> # list of supported modules
> mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
>
> # first check if module is loaded
> -for line in loaded_mods:
> +try:
> +# Get list of syfs modules, some of them might be builtin and merge
> with mods
> +sysfs_path = '/sys/module/'
> +
> +# Get the list of directories in sysfs_path
> +sysfs_mods = [os.path.join(sysfs_path,o) for o in
> os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
> +
> +# Extract the last element of '/sys/module/abc' in the array
> +sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
> +
> +# special case for vfio_pci (module is named vfio-pci,
> +# but its .ko is named vfio_pci)
> +sysfs_mods = map(lambda a: a if a != 'vfio_pci' else 'vfio-pci',
> sysfs_mods)
> +
> for mod in mods:
> -if line.startswith(mod["Name"]):
> -mod["Found"] = True
> -# special case for vfio_pci (module is named vfio-pci,
> -# but its .ko is named vfio_pci)
> -elif line.replace("_", "-").startswith(mod["Name"]):
> +if mod["Found"] == False and (mod["Name"] in sysfs_mods):
> mod["Found"] = True
> +except:
> +pass
>
> # check if we have at least one loaded module
> if True not in [mod["Found"] for mod in mods] and b_flag is not None:
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
ping?
W dniu 16.12.2015 o 15:14, Kamil Rytarowski pisze:
> ping?
>
> W dniu 09.12.2015 o 14:19, Kamil Rytarowski pisze:
>> Currently dpdk_nic_bind.py detects Linux kernel modules via reading
>> /proc/modules. Built-in ones aren't listed there and therefore they
>> are not
>> being found by the script.
>>
>> Add support for checking built-in modules with parsing the sysfs files.
>>
>> This commit obsoletes the /proc/modules parsing approach.
>>
>> Signed-off-by: Kamil Rytarowski
>> Signed-off-by: David Marchand
>> ---
>> tools/dpdk_nic_bind.py | 27 +--
>> 1 file changed, 17 insertions(+), 10 deletions(-)
>>
>> diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
>> index f02454e..e161062 100755
>> --- a/tools/dpdk_nic_bind.py
>> +++ b/tools/dpdk_nic_bind.py
>> @@ -156,22 +156,29 @@ def check_modules():
>> '''Checks that igb_uio is loaded'''
>> global dpdk_drivers
>> -fd = file("/proc/modules")
>> -loaded_mods = fd.readlines()
>> -fd.close()
>> -
>> # list of supported modules
>> mods = [{"Name" : driver, "Found" : False} for driver in
>> dpdk_drivers]
>> # first check if module is loaded
>> -for line in loaded_mods:
>> +try:
>> +# Get list of syfs modules, some of them might be builtin
>> and merge with mods
>> +sysfs_path = '/sys/module/'
>> +
>> +# Get the list of directories in sysfs_path
>> +sysfs_mods = [os.path.join(sysfs_path,o) for o in
>> os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
>> +
>> +# Extract the last element of '/sys/module/abc' in the array
>> +sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
>> +
>> +# special case for vfio_pci (module is named vfio-pci,
>> +# but its .ko is named vfio_pci)
>> +sysfs_mods = map(lambda a: a if a != 'vfio_pci' else
>> 'vfio-pci', sysfs_mods)
>> +
>> for mod in mods:
>> -if line.startswith(mod["Name"]):
>> -mod["Found"] = True
>> -# special case for vfio_pci (module is named vfio-pci,
>> -# but its .ko is named vfio_pci)
>> -elif line.replace("_", "-").startswith(mod["Name"]):
>> +if mod["Found"] == False and (mod["Name"] in sysfs_mods):
>> mod["Found"] = True
>> +except:
>> +pass
>> # check if we have at least one loaded module
>> if True not in [mod["Found"] for mod in mods] and b_flag is not
>> None:
>
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
Thank you, I will submit improved version as v6.
W dniu 18.01.2016 o 15:21, Yuanhan Liu pisze:
> Hi Kamil,
>
> First of all, sorry for no one has reviewed your patches for over one
> month! You may want to ping more often (say, per week) next time if it
> still happenes :)
>
> Another thing is that there is no maintainer for tools code.
>
> On Wed, Dec 09, 2015 at 02:19:57PM +0100, Kamil Rytarowski wrote:
>> Currently dpdk_nic_bind.py detects Linux kernel modules via reading
>> /proc/modules. Built-in ones aren't listed there and therefore they are not
>> being found by the script.
>>
>> Add support for checking built-in modules with parsing the sysfs files.
>>
>> This commit obsoletes the /proc/modules parsing approach.
>>
>> Signed-off-by: Kamil Rytarowski
>> Signed-off-by: David Marchand
>> ---
>> tools/dpdk_nic_bind.py | 27 +--
>> 1 file changed, 17 insertions(+), 10 deletions(-)
>>
>> diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
>> index f02454e..e161062 100755
>> --- a/tools/dpdk_nic_bind.py
>> +++ b/tools/dpdk_nic_bind.py
>> @@ -156,22 +156,29 @@ def check_modules():
>> '''Checks that igb_uio is loaded'''
>> global dpdk_drivers
>>
>> -fd = file("/proc/modules")
>> -loaded_mods = fd.readlines()
>> -fd.close()
>> -
>> # list of supported modules
>> mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
>>
>> # first check if module is loaded
>> -for line in loaded_mods:
>> +try:
>> +# Get list of syfs modules, some of them might be builtin and merge
>> with mods
>> +sysfs_path = '/sys/module/'
>> +
>> +# Get the list of directories in sysfs_path
>> +sysfs_mods = [os.path.join(sysfs_path,o) for o in
>> os.listdir(sysfs_path) if os.path.isdir(os.path.join(sysfs_path,o))]
> Minor nit: it's quite a long line; you may need break it. And space is
> needed after ','.
>
> Otherwise, this patch looks good to me.
>
> --yliu
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
W dniu 18.01.2016 o 15:32, Thomas Monjalon pisze: > Hi Kamil, > > 2015-12-09 14:19, Kamil Rytarowski: >> Currently dpdk_nic_bind.py detects Linux kernel modules via reading >> /proc/modules. Built-in ones aren't listed there and therefore they are not >> being found by the script. >> >> Add support for checking built-in modules with parsing the sysfs files. >> >> This commit obsoletes the /proc/modules parsing approach. >> >> Signed-off-by: Kamil Rytarowski > I have a doubt about this tag: >> Signed-off-by: David Marchand > What do you mean here? Excuse me, it should be: Acked-by: David Marchand http://dpdk.org/ml/archives/dev/2015-December/029720.html
[dpdk-dev] [PATCH v5 2/2] eal/linux: Add support for handling built-in kernel modules
W dniu 18.01.2016 o 15:22, Yuanhan Liu pisze: > On Wed, Dec 09, 2015 at 02:19:58PM +0100, Kamil Rytarowski wrote: >> Currently rte_eal_check_module() detects Linux kernel modules via reading >> /proc/modules. Built-in ones aren't listed there and therefore they are not >> being found by the script. >> >> Add support for checking built-in modules with parsing the sysfs files >> >> This commit obsoletes the /proc/modules parsing approach. >> >> Signed-off-by: Kamil Rytarowski >> Signed-off-by: David Marchand > Acked-by: Yuanhan Liu > > Thanks. > > --yliu Thank you, I will submit v6 and note you with the the Acked-by line.
[dpdk-dev] [PATCH v6 1/2] tools: Add support for handling built-in kernel modules
ping?
W dniu 20.01.2016 o 10:48, krytarowski at caviumnetworks.com pisze:
> From: Kamil Rytarowski
>
> Currently dpdk_nic_bind.py detects Linux kernel modules via reading
> /proc/modules. Built-in ones aren't listed there and therefore they are not
> being found by the script.
>
> Add support for checking built-in modules with parsing the sysfs files.
>
> This commit obsoletes the /proc/modules parsing approach.
>
> Signed-off-by: Kamil Rytarowski
> Acked-by: David Marchand
> Acked-by: Yuanhan Liu
> ---
> tools/dpdk_nic_bind.py | 30 --
> 1 file changed, 20 insertions(+), 10 deletions(-)
>
> diff --git a/tools/dpdk_nic_bind.py b/tools/dpdk_nic_bind.py
> index f02454e..1d16d9f 100755
> --- a/tools/dpdk_nic_bind.py
> +++ b/tools/dpdk_nic_bind.py
> @@ -156,22 +156,32 @@ def check_modules():
> '''Checks that igb_uio is loaded'''
> global dpdk_drivers
>
> -fd = file("/proc/modules")
> -loaded_mods = fd.readlines()
> -fd.close()
> -
> # list of supported modules
> mods = [{"Name" : driver, "Found" : False} for driver in dpdk_drivers]
>
> # first check if module is loaded
> -for line in loaded_mods:
> +try:
> +# Get list of syfs modules, some of them might be builtin and merge
> with mods
> +sysfs_path = '/sys/module/'
> +
> +# Get the list of directories in sysfs_path
> +sysfs_mods = [os.path.join(sysfs_path, o) for o
> + in os.listdir(sysfs_path)
> + if os.path.isdir(os.path.join(sysfs_path, o))]
> +
> +# Extract the last element of '/sys/module/abc' in the array
> +sysfs_mods = [a.split('/')[-1] for a in sysfs_mods]
> +
> +# special case for vfio_pci (module is named vfio-pci,
> +# but its .ko is named vfio_pci)
> +sysfs_mods = map(lambda a:
> + a if a != 'vfio_pci' else 'vfio-pci', sysfs_mods)
> +
> for mod in mods:
> -if line.startswith(mod["Name"]):
> -mod["Found"] = True
> -# special case for vfio_pci (module is named vfio-pci,
> -# but its .ko is named vfio_pci)
> -elif line.replace("_", "-").startswith(mod["Name"]):
> +if mod["Found"] == False and (mod["Name"] in sysfs_mods):
> mod["Found"] = True
> +except:
> +pass
>
> # check if we have at least one loaded module
> if True not in [mod["Found"] for mod in mods] and b_flag is not None:
[dpdk-dev] [PATCH v5 1/2] tools: Add support for handling built-in kernel modules
W dniu 26.01.2016 o 16:12, Thomas Monjalon pisze: > 2016-01-19 17:35, Kamil Rytarowski: >> W dniu 18.01.2016 o 15:32, Thomas Monjalon pisze: >>> Hi Kamil, >>> >>> 2015-12-09 14:19, Kamil Rytarowski: >>>> Currently dpdk_nic_bind.py detects Linux kernel modules via reading >>>> /proc/modules. Built-in ones aren't listed there and therefore they are not >>>> being found by the script. >>>> >>>> Add support for checking built-in modules with parsing the sysfs files. >>>> >>>> This commit obsoletes the /proc/modules parsing approach. >>>> >>>> Signed-off-by: Kamil Rytarowski >>> I have a doubt about this tag: >>>> Signed-off-by: David Marchand >>> What do you mean here? >> Excuse me, it should be: Acked-by: David Marchand >> >> >> http://dpdk.org/ml/archives/dev/2015-December/029720.html > The ack was only for the patch 2/2 I see. I will correct it.
[dpdk-dev] [PATCH v6 1/2] tools: Add support for handling built-in kernel modules
W dniu 26.01.2016 o 16:23, Thomas Monjalon pisze: > 2016-01-20 10:48, krytarowski at caviumnetworks.com: >> --- a/tools/dpdk_nic_bind.py >> +++ b/tools/dpdk_nic_bind.py >> -for line in loaded_mods: >> +try: >> +# Get list of syfs modules, some of them might be builtin and merge >> with mods > Please could you explain this comment? > Is it remaining from previous versions of the patch? Yes. It might be changed to: # Get list of sysfs modules (both built-in and dynamically loaded) > [...] >> +# special case for vfio_pci (module is named vfio-pci, >> +# but its .ko is named vfio_pci) > Isn't it common to have dash replaced by underscore for kernel modules? > I retained the logic for special case of vfio-pci. At the moment (according to my knowledge) there are no other DPDK modules with this name replacement. I checked few example Linux modules and if a module is named with dash, it's being replaced to underscore. The modprobe(8) tool can accept both names as interchangeable (with dash and underscore). Would you like to make it a general rule and replace all dashes with underscores? Thank you
[dpdk-dev] [PATCH v6 1/2] tools: Add support for handling built-in kernel modules
W dniu 28.01.2016 o 12:22, Panu Matilainen pisze: > On 01/28/2016 01:17 PM, Kamil Rytarowski wrote: >> I retained the logic for special case of vfio-pci. At the moment >> (according to my knowledge) there are no other DPDK modules with this >> name replacement. >> >> I checked few example Linux modules and if a module is named with dash, >> it's being replaced to underscore. The modprobe(8) tool can accept both >> names as interchangeable (with dash and underscore). >> >> Would you like to make it a general rule and replace all dashes with >> underscores? > > It would be nice to behave the same as modprobe wrt dash and > underscore, yes. > > - Panu - > My patch is intended to support built-in modules, the rest isn't that trivial without changing the behavior. I prototyped it and it added extra unnecessary complexity, while we just want to handle vfio_pci -> vfio-pci. I'm going to submit new version with improved comment in the code. Please continue possible improvements in separate threads.
[dpdk-dev] [PATCH v6 1/2] tools: Add support for handling built-in kernel modules
W dniu 28.01.2016 o 14:52, Thomas Monjalon pisze: > 2016-01-28 12:17, Kamil Rytarowski: > [...] >>>> +# special case for vfio_pci (module is named vfio-pci, >>>> +# but its .ko is named vfio_pci) >>> Isn't it common to have dash replaced by underscore for kernel modules? >>> >> I retained the logic for special case of vfio-pci. At the moment >> (according to my knowledge) there are no other DPDK modules with this >> name replacement. >> >> I checked few example Linux modules and if a module is named with dash, >> it's being replaced to underscore. The modprobe(8) tool can accept both >> names as interchangeable (with dash and underscore). >> >> Would you like to make it a general rule and replace all dashes with >> underscores? > I don't know. Do what you think is best. > Thanks I suggest to leave it as it is. There is no need to add a layer of abstraction for a single known exception. Also, this is out of scope of my patch. Thank you.
[dpdk-dev] [PATCH 00/13] Add support for secondary queue set in nicvf thunderx driver
This series of patches adds support for secondary queue set in nicvf thunderx driver There are two types of VFs: - Primary VF - Secondary VF Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. In case port is configured to use more than 8 queues, then it requires one (or more) secondary VF. Each secondary VF adds additional 8 queues to the queue set. During PMD driver initialization, the primary VF's are enumerated by checking the specific flag (see READY message). They are at the beginning of VF list (the remain ones are secondary VF's). The primary VFs are used as master queue sets. Secondary VFs provide additional queue sets for primary ones. If a port is configured for more then 8 queues then it will request for additional queues from secondary VFs. Secondary VFs cannot be shared between primary VFs. Primary VFs are present on the tail of the 'Network devices using kernel driver' list, secondary VFs are on the remaining tail of the list. The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attach with an id below or equal to this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. This patchset also contains other cleanups and improvements like fixing erroneous checksum calculation and preparing the thunderx driver for the multi queue set feature support. These changes base on the following pending patches: [dpdk-dev,1/3] net/thunderx: remove generic passx references from the driver http://dpdk.org/dev/patchwork/patch/14963/ [dpdk-dev,2/3] net/thunderx: introduce cqe_rx2 HW capability flag http://dpdk.org/dev/patchwork/patch/14964/ [dpdk-dev,3/3] net/thunderx: add 81xx SoC support http://dpdk.org/dev/patchwork/patch/14965/ Kamil Rytarowski (13): net/thunderx: cleanup the driver before adding new features net/thunderx: correct transmit checksum handling net/thunderx/base: add family of functions to store qsets net/thunderx/base: add secondary queue set support net/thunderx: add family of functions to store DPDK qsets net/thunderx: add secondary queue set in interrupt functions net/thunderx: fix multiprocess support in stats net/thunderx: add helper utils for secondary qset support net/thunderx: add secondary qset support in dev stop/close net/thunderx: add secondary qset support in device start net/thunderx: add secondary qset support in device configure net/thunderx: add final bits for secondary queue support net/thunderx: document secondary queue set support doc/guides/nics/thunderx.rst | 114 - drivers/net/thunderx/Makefile | 2 + drivers/net/thunderx/base/nicvf_bsvf.c| 72 +++ drivers/net/thunderx/base/nicvf_bsvf.h| 76 +++ drivers/net/thunderx/base/nicvf_hw.c | 10 +- drivers/net/thunderx/base/nicvf_hw.h | 6 +- drivers/net/thunderx/base/nicvf_hw_defs.h | 1 + drivers/net/thunderx/base/nicvf_mbox.c| 34 +- drivers/net/thunderx/base/nicvf_mbox.h| 21 +- drivers/net/thunderx/nicvf_ethdev.c | 753 +- drivers/net/thunderx/nicvf_ethdev.h | 39 ++ drivers/net/thunderx/nicvf_rxtx.c | 14 +- drivers/net/thunderx/nicvf_struct.h | 6 +- drivers/net/thunderx/nicvf_svf.c | 78 drivers/net/thunderx/nicvf_svf.h | 66 +++ 15 files changed, 1046 insertions(+), 246 deletions(-) create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h create mode 100644 drivers/net/thunderx/nicvf_svf.c create mode 100644 drivers/net/thunderx/nicvf_svf.h -- 1.9.1
[dpdk-dev] [PATCH 01/13] net/thunderx: cleanup the driver before adding new features
From: Kamil Rytarowski
Refactored features:
- enable nicvf_qset_rbdr_precharge to handle handle secondary queue sets
- rte_free already handles NULL pointer
- check mempool flags to predict being contiguous in memory
- allow to use mempool with multiple memory chunks
- simplify local construct of accessing nb_rx_queus
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/base/nicvf_hw.c | 10 +-
drivers/net/thunderx/base/nicvf_hw.h | 6 +++---
drivers/net/thunderx/nicvf_ethdev.c | 32
3 files changed, 20 insertions(+), 28 deletions(-)
diff --git a/drivers/net/thunderx/base/nicvf_hw.c
b/drivers/net/thunderx/base/nicvf_hw.c
index 4bdd183..1f08ef2 100644
--- a/drivers/net/thunderx/base/nicvf_hw.c
+++ b/drivers/net/thunderx/base/nicvf_hw.c
@@ -141,7 +141,7 @@ nicvf_base_init(struct nicvf *nic)
return NICVF_ERR_BASE_INIT;
if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN88XX_PASS2_NICVF)
- nic->hwcap |= NICVF_CAP_TUNNEL_PARSING;
+ nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
if (nicvf_hw_version(nic) == PCI_SUB_DEVICE_ID_CN81XX_NICVF)
nic->hwcap |= NICVF_CAP_TUNNEL_PARSING | NICVF_CAP_CQE_RX2;
@@ -497,9 +497,9 @@ nicvf_qsize_rbdr_roundup(uint32_t val)
}
int
-nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
- rbdr_pool_get_handler handler,
- void *opaque, uint32_t max_buffs)
+nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+ uint16_t ridx, rbdr_pool_get_handler handler,
+ uint32_t max_buffs)
{
struct rbdr_entry_t *desc, *desc0;
struct nicvf_rbdr *rbdr = nic->rbdr;
@@ -514,7 +514,7 @@ nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
if (count >= max_buffs)
break;
desc0 = desc + count;
- phy = handler(opaque);
+ phy = handler(dev, nic);
if (phy) {
desc0->full_addr = phy;
count++;
diff --git a/drivers/net/thunderx/base/nicvf_hw.h
b/drivers/net/thunderx/base/nicvf_hw.h
index a6cda82..2b8738b 100644
--- a/drivers/net/thunderx/base/nicvf_hw.h
+++ b/drivers/net/thunderx/base/nicvf_hw.h
@@ -85,7 +85,7 @@ enum nicvf_err_e {
NICVF_ERR_RSS_GET_SZ,/* -8171 */
};
-typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *opaque);
+typedef nicvf_phys_addr_t (*rbdr_pool_get_handler)(void *dev, void *opaque);
struct nicvf_hw_rx_qstats {
uint64_t q_rx_bytes;
@@ -194,8 +194,8 @@ int nicvf_qset_reclaim(struct nicvf *nic);
int nicvf_qset_rbdr_config(struct nicvf *nic, uint16_t qidx);
int nicvf_qset_rbdr_reclaim(struct nicvf *nic, uint16_t qidx);
-int nicvf_qset_rbdr_precharge(struct nicvf *nic, uint16_t ridx,
- rbdr_pool_get_handler handler, void *opaque,
+int nicvf_qset_rbdr_precharge(void *dev, struct nicvf *nic,
+ uint16_t ridx, rbdr_pool_get_handler handler,
uint32_t max_buffs);
int nicvf_qset_rbdr_active(struct nicvf *nic, uint16_t qidx);
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 4402f6a..48f2cd2 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -691,7 +691,7 @@ nicvf_configure_cpi(struct rte_eth_dev *dev)
int ret;
/* Count started rx queues */
- for (qidx = qcnt = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++)
+ for (qidx = qcnt = 0; qidx < dev->data->nb_rx_queues; qidx++)
if (dev->data->rx_queue_state[qidx] ==
RTE_ETH_QUEUE_STATE_STARTED)
qcnt++;
@@ -1023,12 +1023,9 @@ nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t
qidx)
static void
nicvf_dev_rx_queue_release(void *rx_queue)
{
- struct nicvf_rxq *rxq = rx_queue;
-
PMD_INIT_FUNC_TRACE();
- if (rxq)
- rte_free(rxq);
+ rte_free(rx_queue);
}
static int
@@ -1087,9 +1084,9 @@ nicvf_dev_rx_queue_setup(struct rte_eth_dev *dev,
uint16_t qidx,
PMD_DRV_LOG(WARNING, "socket_id expected %d, configured %d",
socket_id, nic->node);
- /* Mempool memory should be contiguous */
- if (mp->nb_mem_chunks != 1) {
- PMD_INIT_LOG(ERR, "Non contiguous mempool, check huge page sz");
+ /* Mempool memory must be contiguous */
+ if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG) {
+ PMD_INIT_LOG(ERR, "Mempool memory must be contiguous");
return -EINVAL;
}
@@ -1212,15 +1
[dpdk-dev] [PATCH 02/13] net/thunderx: correct transmit checksum handling
From: Kamil Rytarowski
The symbols PKT_TX_TCP_CKSUM and PKT_TX_UDP_CKSUM are not bits on a
bitmask. Set l3_offset always for TX offloads, not just for PKT_TX_IP_CKSUM
being true.
Fixes: 1c421f18e095 ("net/thunderx: add single and multi-segment Tx")
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_rxtx.c | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_rxtx.c
b/drivers/net/thunderx/nicvf_rxtx.c
index eb51a72..e15c730 100644
--- a/drivers/net/thunderx/nicvf_rxtx.c
+++ b/drivers/net/thunderx/nicvf_rxtx.c
@@ -70,19 +70,20 @@ fill_sq_desc_header(union sq_entry_t *entry, struct
rte_mbuf *pkt)
ol_flags = pkt->ol_flags & NICVF_TX_OFFLOAD_MASK;
if (unlikely(ol_flags)) {
/* L4 cksum */
- if (ol_flags & PKT_TX_TCP_CKSUM)
+ uint64_t l4_flags = ol_flags & PKT_TX_L4_MASK;
+ if (l4_flags == PKT_TX_TCP_CKSUM)
sqe.hdr.csum_l4 = SEND_L4_CSUM_TCP;
- else if (ol_flags & PKT_TX_UDP_CKSUM)
+ else if (l4_flags == PKT_TX_UDP_CKSUM)
sqe.hdr.csum_l4 = SEND_L4_CSUM_UDP;
else
sqe.hdr.csum_l4 = SEND_L4_CSUM_DISABLE;
+
+ sqe.hdr.l3_offset = pkt->l2_len;
sqe.hdr.l4_offset = pkt->l3_len + pkt->l2_len;
/* L3 cksum */
- if (ol_flags & PKT_TX_IP_CKSUM) {
+ if (ol_flags & PKT_TX_IP_CKSUM)
sqe.hdr.csum_l3 = 1;
- sqe.hdr.l3_offset = pkt->l2_len;
- }
}
entry->buff[0] = sqe.buff[0];
--
1.9.1
[dpdk-dev] [PATCH 03/13] net/thunderx/base: add family of functions to store qsets
From: Kamil Rytarowski
This interface (nicvf_bsvf) will be used for secondary queue set support.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/Makefile | 1 +
drivers/net/thunderx/base/nicvf_bsvf.c | 72
drivers/net/thunderx/base/nicvf_bsvf.h | 76 ++
3 files changed, 149 insertions(+)
create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.c
create mode 100644 drivers/net/thunderx/base/nicvf_bsvf.h
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 8ea6b45..711400a 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -57,6 +57,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_rxtx.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.c
b/drivers/net/thunderx/base/nicvf_bsvf.c
new file mode 100644
index 000..9e028a3
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.c
@@ -0,0 +1,72 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include
+#include
+#include
+
+#include "nicvf_bsvf.h"
+#include "nicvf_plat.h"
+
+static SIMPLEQ_HEAD(, svf_entry) head = SIMPLEQ_HEAD_INITIALIZER(head);
+
+void
+nicvf_bsvf_push(struct svf_entry *entry)
+{
+ assert(entry != NULL);
+ assert(entry->vf != NULL);
+
+ SIMPLEQ_INSERT_TAIL(&head, entry, next);
+}
+
+struct svf_entry *
+nicvf_bsvf_pop(void)
+{
+ struct svf_entry *entry;
+
+ assert(!SIMPLEQ_EMPTY(&head));
+
+ entry = SIMPLEQ_FIRST(&head);
+
+ assert(entry != NULL);
+ assert(entry->vf != NULL);
+
+ SIMPLEQ_REMOVE_HEAD(&head, next);
+
+ return entry;
+}
+
+int
+nicvf_bsvf_empty(void)
+{
+ return SIMPLEQ_EMPTY(&head);
+}
diff --git a/drivers/net/thunderx/base/nicvf_bsvf.h
b/drivers/net/thunderx/base/nicvf_bsvf.h
new file mode 100644
index 000..5d5a25e
--- /dev/null
+++ b/drivers/net/thunderx/base/nicvf_bsvf.h
@@ -0,0 +1,76 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE
[dpdk-dev] [PATCH 04/13] net/thunderx/base: add secondary queue set support
From: Kamil Rytarowski
Changes:
- add new message sqs_alloc in mailbox
- add a queue container to hold secondary qsets.
- add nicvf_mbox_request_sqs
- handle new mailbox messages for secondary queue set support
- register secondary queue sets for furthe reuse
- register the number secondary queue sets in MSG_QS_CFG
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/base/nicvf_hw_defs.h | 1 +
drivers/net/thunderx/base/nicvf_mbox.c| 34 ++-
drivers/net/thunderx/base/nicvf_mbox.h| 21 +--
drivers/net/thunderx/nicvf_struct.h | 5 +
4 files changed, 58 insertions(+), 3 deletions(-)
diff --git a/drivers/net/thunderx/base/nicvf_hw_defs.h
b/drivers/net/thunderx/base/nicvf_hw_defs.h
index 2f2b225..3b947e0 100644
--- a/drivers/net/thunderx/base/nicvf_hw_defs.h
+++ b/drivers/net/thunderx/base/nicvf_hw_defs.h
@@ -207,6 +207,7 @@
#define NICVF_CQE_RX2_RBPTR_WORD(7)
#define NICVF_STATIC_ASSERT(s) _Static_assert(s, #s)
+#define assert_if_secondary(nic) assert((nic)->sqs_mode == 0)
typedef uint64_t nicvf_phys_addr_t;
diff --git a/drivers/net/thunderx/base/nicvf_mbox.c
b/drivers/net/thunderx/base/nicvf_mbox.c
index 9c5cd83..974ce97 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.c
+++ b/drivers/net/thunderx/base/nicvf_mbox.c
@@ -62,6 +62,9 @@ static const char *mbox_message[NIC_MBOX_MSG_MAX] = {
[NIC_MBOX_MSG_RESET_STAT_COUNTER] = "NIC_MBOX_MSG_RESET_STAT_COUNTER",
[NIC_MBOX_MSG_CFG_DONE] = "NIC_MBOX_MSG_CFG_DONE",
[NIC_MBOX_MSG_SHUTDOWN] = "NIC_MBOX_MSG_SHUTDOWN",
+ [NIC_MBOX_MSG_RES_BIT]= "NIC_MBOX_MSG_RES_BIT",
+ [NIC_MBOX_MSG_RSS_SIZE_RES_BIT] = "NIC_MBOX_MSG_RSS_SIZE",
+ [NIC_MBOX_MSG_ALLOC_SQS_RES_BIT] = "NIC_MBOX_MSG_ALLOC_SQS",
};
static inline const char * __attribute__((unused))
@@ -173,7 +176,7 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
case NIC_MBOX_MSG_NACK:
nic->pf_nacked = true;
break;
- case NIC_MBOX_MSG_RSS_SIZE:
+ case NIC_MBOX_MSG_RSS_SIZE_RES_BIT:
nic->rss_info.rss_size = mbx.rss_size.ind_tbl_size;
nic->pf_acked = true;
break;
@@ -183,6 +186,13 @@ nicvf_handle_mbx_intr(struct nicvf *nic)
nic->speed = mbx.link_status.speed;
nic->pf_acked = true;
break;
+ case NIC_MBOX_MSG_ALLOC_SQS_RES_BIT:
+ assert_if_secondary(nic);
+ assert(mbx.sqs_alloc.qs_count == nic->sqs_count);
+ for (i = 0; i < mbx.sqs_alloc.qs_count; i++)
+ assert(mbx.sqs_alloc.svf[i] == nic->snicvf[i]->vf_id);
+ nic->pf_acked = true;
+ break;
default:
nicvf_log_error("Invalid message from PF, msg_id=0x%hhx %s",
mbx.msg.msg, nicvf_mbox_msg_str(mbx.msg.msg));
@@ -314,11 +324,33 @@ nicvf_mbox_qset_config(struct nicvf *nic, struct
pf_qs_cfg *qs_cfg)
/* Send a mailbox msg to PF to config Qset */
mbx.msg.msg = NIC_MBOX_MSG_QS_CFG;
mbx.qs.num = nic->vf_id;
+ mbx.qs.sqs_count = nic->sqs_count;
mbx.qs.cfg = qs_cfg->value;
return nicvf_mbox_send_msg_to_pf(nic, &mbx);
}
int
+nicvf_mbox_request_sqs(struct nicvf *nic)
+{
+ struct nic_mbx mbx = { .msg = { 0 } };
+ size_t i;
+
+ assert_if_secondary(nic);
+ assert(nic->sqs_count > 0);
+ assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+ mbx.sqs_alloc.msg = NIC_MBOX_MSG_ALLOC_SQS;
+ mbx.sqs_alloc.spec = 1;
+ mbx.sqs_alloc.qs_count = nic->sqs_count;
+
+ /* Set no of Rx/Tx queues in each of the SQsets */
+ for (i = 0; i < nic->sqs_count; i++)
+ mbx.sqs_alloc.svf[i] = nic->snicvf[i]->vf_id;
+
+ return nicvf_mbox_send_msg_to_pf(nic, &mbx);
+}
+
+int
nicvf_mbox_rq_drop_config(struct nicvf *nic, uint16_t qidx, bool enable)
{
struct nic_mbx mbx = { .msg = { 0 } };
diff --git a/drivers/net/thunderx/base/nicvf_mbox.h
b/drivers/net/thunderx/base/nicvf_mbox.h
index 7c0c6a9..084f3a7 100644
--- a/drivers/net/thunderx/base/nicvf_mbox.h
+++ b/drivers/net/thunderx/base/nicvf_mbox.h
@@ -36,6 +36,7 @@
#include
#include "nicvf_plat.h"
+#include "../nicvf_struct.h"
/* PF <--> VF Mailbox communication
* Two 64bit registers are shared between PF and VF for each VF
@@ -67,10 +68,16 @@
#defineNIC_MBOX_MSG_ALLOC_SQS 0x12/* Allocate secondary
Qset */
#defineNIC_MBOX_MSG_LOOPBACK 0x16/* Set interface in
loopback */
#defineNIC_MBOX_MSG_RESET_STA
[dpdk-dev] [PATCH 05/13] net/thunderx: add family of functions to store DPDK qsets
From: Kamil Rytarowski
These functions (nicvf_svf) are DPDK specialization of base/nicvf_bsvf.[ch]
ones.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/Makefile| 1 +
drivers/net/thunderx/nicvf_svf.c | 78
drivers/net/thunderx/nicvf_svf.h | 66 ++
3 files changed, 145 insertions(+)
create mode 100644 drivers/net/thunderx/nicvf_svf.c
create mode 100644 drivers/net/thunderx/nicvf_svf.h
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 711400a..bcab5f9 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -58,6 +58,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_hw.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_mbox.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_ethdev.c
SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_bsvf.c
+SRCS-$(CONFIG_RTE_LIBRTE_THUNDERX_NICVF_PMD) += nicvf_svf.c
ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
CFLAGS_nicvf_rxtx.o += -fno-prefetch-loop-arrays
diff --git a/drivers/net/thunderx/nicvf_svf.c b/drivers/net/thunderx/nicvf_svf.c
new file mode 100644
index 000..f746e94
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.c
@@ -0,0 +1,78 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include
+#include
+
+#include
+#include
+
+#include "base/nicvf_bsvf.h"
+
+#include "nicvf_svf.h"
+
+void
+nicvf_svf_push(struct nicvf *vf)
+{
+ struct svf_entry *entry = NULL;
+
+ assert(vf != NULL);
+
+ entry = rte_zmalloc("nicvf", sizeof(*entry), RTE_CACHE_LINE_SIZE);
+ if (entry == NULL)
+ rte_panic("Cannoc allocate memory for svf_entry\n");
+
+ entry->vf = vf;
+
+ nicvf_bsvf_push(entry);
+}
+
+struct nicvf *
+nicvf_svf_pop(void)
+{
+ struct nicvf *vf;
+ struct svf_entry *entry;
+
+ entry = nicvf_bsvf_pop();
+
+ vf = entry->vf;
+
+ rte_free(entry);
+
+ return vf;
+}
+
+int
+nicvf_svf_empty(void)
+{
+ return nicvf_bsvf_empty();
+}
diff --git a/drivers/net/thunderx/nicvf_svf.h b/drivers/net/thunderx/nicvf_svf.h
new file mode 100644
index 000..6471aa5
--- /dev/null
+++ b/drivers/net/thunderx/nicvf_svf.h
@@ -0,0 +1,66 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS A
[dpdk-dev] [PATCH 06/13] net/thunderx: add secondary queue set in interrupt functions
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 41 ++---
1 file changed, 24 insertions(+), 17 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 48f2cd2..55f6480 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -101,31 +101,40 @@ nicvf_set_eth_link_status(struct nicvf *nic, struct
rte_eth_link *link)
static void
nicvf_interrupt(void *arg)
{
- struct nicvf *nic = arg;
+ struct rte_eth_dev *dev = arg;
+ struct nicvf *nic = nicvf_pmd_priv(dev);
if (nicvf_reg_poll_interrupts(nic) == NIC_MBOX_MSG_BGX_LINK_CHANGE) {
- if (nic->eth_dev->data->dev_conf.intr_conf.lsc)
- nicvf_set_eth_link_status(nic,
- &nic->eth_dev->data->dev_link);
- _rte_eth_dev_callback_process(nic->eth_dev,
- RTE_ETH_EVENT_INTR_LSC);
+ if (dev->data->dev_conf.intr_conf.lsc)
+ nicvf_set_eth_link_status(nic, &dev->data->dev_link);
+ _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC);
}
rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
- nicvf_interrupt, nic);
+ nicvf_interrupt, dev);
+}
+
+static void __rte_unused
+nicvf_vf_interrupt(void *arg)
+{
+ struct nicvf *nic = arg;
+
+ nicvf_reg_poll_interrupts(nic);
+
+ rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
+ nicvf_vf_interrupt, nic);
}
static int
-nicvf_periodic_alarm_start(struct nicvf *nic)
+nicvf_periodic_alarm_start(void (fn)(void *), void *arg)
{
- return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000,
- nicvf_interrupt, nic);
+ return rte_eal_alarm_set(NICVF_INTR_POLL_INTERVAL_MS * 1000, fn, arg);
}
static int
-nicvf_periodic_alarm_stop(struct nicvf *nic)
+nicvf_periodic_alarm_stop(void (fn)(void *), void *arg)
{
- return rte_eal_alarm_cancel(nicvf_interrupt, nic);
+ return rte_eal_alarm_cancel(fn, arg);
}
/*
@@ -1512,12 +1521,10 @@ nicvf_dev_stop(struct rte_eth_dev *dev)
static void
nicvf_dev_close(struct rte_eth_dev *dev)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
-
PMD_INIT_FUNC_TRACE();
nicvf_dev_stop(dev);
- nicvf_periodic_alarm_stop(nic);
+ nicvf_periodic_alarm_stop(nicvf_interrupt, dev);
}
static int
@@ -1668,7 +1675,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
nicvf_disable_all_interrupts(nic);
- ret = nicvf_periodic_alarm_start(nic);
+ ret = nicvf_periodic_alarm_start(nicvf_interrupt, eth_dev);
if (ret) {
PMD_INIT_LOG(ERR, "Failed to start period alarm");
goto fail;
@@ -1729,7 +1736,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev)
malloc_fail:
rte_free(eth_dev->data->mac_addrs);
alarm_fail:
- nicvf_periodic_alarm_stop(nic);
+ nicvf_periodic_alarm_stop(nicvf_interrupt, eth_dev);
fail:
return ret;
}
--
1.9.1
[dpdk-dev] [PATCH 07/13] net/thunderx: fix multiprocess support in stats
From: Kamil Rytarowski
In case of the multiprocess mode a shared nicvf struct between processes
cannot point with the eth_dev pointer to master device, therefore remove it
allong with references to it refactoring the code where needed.
Fixes: 7413feee662d ("net/thunderx: add device start/stop and close")
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 69 ++---
drivers/net/thunderx/nicvf_rxtx.c | 3 +-
drivers/net/thunderx/nicvf_struct.h | 1 -
3 files changed, 36 insertions(+), 37 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 55f6480..b4ab505 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -497,14 +497,14 @@ nicvf_dev_rss_hash_update(struct rte_eth_dev *dev,
}
static int
-nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq *rxq, uint16_t qidx,
- uint32_t desc_cnt)
+nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ struct nicvf_rxq *rxq, uint16_t qidx, uint32_t desc_cnt)
{
const struct rte_memzone *rz;
uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "cq_ring", qidx, ring_size,
- NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+ NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
return -ENOMEM;
@@ -520,13 +520,13 @@ nicvf_qset_cq_alloc(struct nicvf *nic, struct nicvf_rxq
*rxq, uint16_t qidx,
}
static int
-nicvf_qset_sq_alloc(struct nicvf *nic, struct nicvf_txq *sq, uint16_t qidx,
- uint32_t desc_cnt)
+nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ struct nicvf_txq *sq, uint16_t qidx, uint32_t desc_cnt)
{
const struct rte_memzone *rz;
uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "sq", qidx, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
@@ -543,7 +543,8 @@ nicvf_qset_sq_alloc(struct nicvf *nic, struct nicvf_txq
*sq, uint16_t qidx,
}
static int
-nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt, uint32_t buffsz)
+nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint32_t desc_cnt, uint32_t buffsz)
{
struct nicvf_rbdr *rbdr;
const struct rte_memzone *rz;
@@ -558,7 +559,7 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t desc_cnt,
uint32_t buffsz)
}
ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
- rz = rte_eth_dma_zone_reserve(nic->eth_dev, "rbdr", 0, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
@@ -583,14 +584,15 @@ nicvf_qset_rbdr_alloc(struct nicvf *nic, uint32_t
desc_cnt, uint32_t buffsz)
}
static void
-nicvf_rbdr_release_mbuf(struct nicvf *nic, nicvf_phys_addr_t phy)
+nicvf_rbdr_release_mbuf(struct rte_eth_dev *dev, struct nicvf *nic
__rte_unused,
+ nicvf_phys_addr_t phy)
{
uint16_t qidx;
void *obj;
struct nicvf_rxq *rxq;
- for (qidx = 0; qidx < nic->eth_dev->data->nb_rx_queues; qidx++) {
- rxq = nic->eth_dev->data->rx_queues[qidx];
+ for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+ rxq = dev->data->rx_queues[qidx];
if (rxq->precharge_cnt) {
obj = (void *)nicvf_mbuff_phy2virt(phy,
rxq->mbuf_phys_off);
@@ -602,7 +604,7 @@ nicvf_rbdr_release_mbuf(struct nicvf *nic,
nicvf_phys_addr_t phy)
}
static inline void
-nicvf_rbdr_release_mbufs(struct nicvf *nic)
+nicvf_rbdr_release_mbufs(struct rte_eth_dev *dev, struct nicvf *nic)
{
uint32_t qlen_mask, head;
struct rbdr_entry_t *entry;
@@ -612,7 +614,7 @@ nicvf_rbdr_release_mbufs(struct nicvf *nic)
head = rbdr->head;
while (head != rbdr->tail) {
entry = rbdr->desc + head;
-
[dpdk-dev] [PATCH 08/13] net/thunderx: add helper utils for secondary qset support
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.h | 39 +
1 file changed, 39 insertions(+)
diff --git a/drivers/net/thunderx/nicvf_ethdev.h
b/drivers/net/thunderx/nicvf_ethdev.h
index 34447e0..e162364 100644
--- a/drivers/net/thunderx/nicvf_ethdev.h
+++ b/drivers/net/thunderx/nicvf_ethdev.h
@@ -87,6 +87,17 @@ nicvf_mbuff_meta_length(struct rte_mbuf *mbuf)
return (uint16_t)((uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf);
}
+static inline uint16_t
+nicvf_netdev_qidx(struct nicvf *nic, uint8_t local_qidx)
+{
+ uint16_t global_qidx = local_qidx;
+
+ if (nic->sqs_mode)
+ global_qidx += ((nic->sqs_id + 1) * MAX_CMP_QUEUES_PER_QS);
+
+ return global_qidx;
+}
+
/*
* Simple phy2virt functions assuming mbufs are in a single huge page
* V = P + offset
@@ -104,4 +115,32 @@ nicvf_mbuff_virt2phy(uintptr_t virt, uint64_t
mbuf_phys_off)
return (phys_addr_t)(virt - mbuf_phys_off);
}
+static inline void
+nicvf_tx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *tx_start,
+ uint16_t *tx_end)
+{
+ uint16_t tmp;
+
+ *tx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+ MAX_SND_QUEUES_PER_QS);
+ tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+MAX_SND_QUEUES_PER_QS) - 1;
+ *tx_end = dev->data->nb_tx_queues ?
+ RTE_MIN(tmp, dev->data->nb_tx_queues - 1) : 0;
+}
+
+static inline void
+nicvf_rx_range(struct rte_eth_dev *dev, struct nicvf *nic, uint16_t *rx_start,
+ uint16_t *rx_end)
+{
+ uint16_t tmp;
+
+ *rx_start = RTE_ALIGN_FLOOR(nicvf_netdev_qidx(nic, 0),
+ MAX_RCV_QUEUES_PER_QS);
+ tmp = RTE_ALIGN_CEIL(nicvf_netdev_qidx(nic, 0) + 1,
+MAX_RCV_QUEUES_PER_QS) - 1;
+ *rx_end = dev->data->nb_rx_queues ?
+ RTE_MIN(tmp, dev->data->nb_rx_queues - 1) : 0;
+}
+
#endif /* __THUNDERX_NICVF_ETHDEV_H__ */
--
1.9.1
[dpdk-dev] [PATCH 09/13] net/thunderx: add secondary qset support in dev stop/close
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 142 +++-
1 file changed, 107 insertions(+), 35 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index b4ab505..c9c5b78 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -67,9 +67,13 @@
#include "nicvf_ethdev.h"
#include "nicvf_rxtx.h"
+#include "nicvf_svf.h"
#include "nicvf_logs.h"
static void nicvf_dev_stop(struct rte_eth_dev *dev);
+static void nicvf_dev_stop_cleanup(struct rte_eth_dev *dev, bool cleanup);
+static void nicvf_vf_stop(struct rte_eth_dev *dev, struct nicvf *nic,
+ bool cleanup);
static inline int
nicvf_atomic_write_link_status(struct rte_eth_dev *dev,
@@ -674,23 +678,29 @@ config_sq_error:
}
static inline int
-nicvf_stop_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
struct nicvf_txq *txq;
int ret;
- if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+ assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+ if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STOPPED)
return 0;
- ret = nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+ ret = nicvf_qset_sq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim sq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim sq VF%d %d %d",
+nic->vf_id, qidx, ret);
- txq = dev->data->tx_queues[qidx];
+ txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
nicvf_tx_queue_release_mbufs(txq);
nicvf_tx_queue_reset(txq);
- dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+ dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
return ret;
}
@@ -1002,30 +1012,34 @@ config_rq_error:
}
static inline int
-nicvf_stop_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_stop_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
struct nicvf_rxq *rxq;
int ret, other_error;
- if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STOPPED)
+ if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STOPPED)
return 0;
ret = nicvf_qset_rq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim rq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim rq VF%d %d %d",
+nic->vf_id, qidx, ret);
other_error = ret;
- rxq = dev->data->rx_queues[qidx];
+ rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
nicvf_rx_queue_release_mbufs(dev, rxq);
nicvf_rx_queue_reset(rxq);
ret = nicvf_qset_cq_reclaim(nic, qidx);
if (ret)
- PMD_INIT_LOG(ERR, "Failed to reclaim cq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to reclaim cq VF%d %d %d",
+nic->vf_id, qidx, ret);
other_error |= ret;
- dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STOPPED;
+ dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
return other_error;
}
@@ -1057,8 +1071,14 @@ static int
nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
{
int ret;
+ struct nicvf *nic = nicvf_pmd_priv(dev);
- ret = nicvf_stop_rx_queue(dev, qidx);
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+ ret = nicvf_vf_stop_rx_queue(dev, nic, qidx);
ret |= nicvf_configure_cpi(dev);
ret |= nicvf_configure_rss_reta(dev);
return ret;
@@ -1073,7 +1093,14 @@ nicvf_dev_tx_queue_start(struct rte_eth_dev *dev,
uint16_t qidx)
static int
nicvf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t qidx)
{
- return nicvf_stop_tx_queue(dev, qidx);
+ struct nicvf *nic = nicvf_pmd_priv(dev);
+
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+ return nicvf_
[dpdk-dev] [PATCH 10/13] net/thunderx: add secondary qset support in device start
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 266 +++-
1 file changed, 172 insertions(+), 94 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index c9c5b78..8510675 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -653,27 +653,33 @@ nicvf_tx_queue_reset(struct nicvf_txq *txq)
}
static inline int
-nicvf_start_tx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_tx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
struct nicvf_txq *txq;
int ret;
- if (dev->data->tx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+ assert(qidx < MAX_SND_QUEUES_PER_QS);
+
+ if (dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STARTED)
return 0;
- txq = dev->data->tx_queues[qidx];
+ txq = dev->data->tx_queues[nicvf_netdev_qidx(nic, qidx)];
txq->pool = NULL;
- ret = nicvf_qset_sq_config(nicvf_pmd_priv(dev), qidx, txq);
+ ret = nicvf_qset_sq_config(nic, qidx, txq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure sq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure sq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_sq_error;
}
- dev->data->tx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+ dev->data->tx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STARTED;
return ret;
config_sq_error:
- nicvf_qset_sq_reclaim(nicvf_pmd_priv(dev), qidx);
+ nicvf_qset_sq_reclaim(nic, qidx);
return ret;
}
@@ -977,31 +983,37 @@ nicvf_rx_queue_reset(struct nicvf_rxq *rxq)
}
static inline int
-nicvf_start_rx_queue(struct rte_eth_dev *dev, uint16_t qidx)
+nicvf_vf_start_rx_queue(struct rte_eth_dev *dev, struct nicvf *nic,
+ uint16_t qidx)
{
- struct nicvf *nic = nicvf_pmd_priv(dev);
struct nicvf_rxq *rxq;
int ret;
- if (dev->data->rx_queue_state[qidx] == RTE_ETH_QUEUE_STATE_STARTED)
+ assert(qidx < MAX_RCV_QUEUES_PER_QS);
+
+ if (dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] ==
+ RTE_ETH_QUEUE_STATE_STARTED)
return 0;
/* Update rbdr pointer to all rxq */
- rxq = dev->data->rx_queues[qidx];
+ rxq = dev->data->rx_queues[nicvf_netdev_qidx(nic, qidx)];
rxq->shared_rbdr = nic->rbdr;
ret = nicvf_qset_rq_config(nic, qidx, rxq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure rq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure rq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_rq_error;
}
ret = nicvf_qset_cq_config(nic, qidx, rxq);
if (ret) {
- PMD_INIT_LOG(ERR, "Failed to configure cq %d %d", qidx, ret);
+ PMD_INIT_LOG(ERR, "Failed to configure cq VF%d %d %d",
+nic->vf_id, qidx, ret);
goto config_cq_error;
}
- dev->data->rx_queue_state[qidx] = RTE_ETH_QUEUE_STATE_STARTED;
+ dev->data->rx_queue_state[nicvf_netdev_qidx(nic, qidx)] =
+ RTE_ETH_QUEUE_STATE_STARTED;
return 0;
config_cq_error:
@@ -1054,9 +1066,15 @@ nicvf_dev_rx_queue_release(void *rx_queue)
static int
nicvf_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
{
+ struct nicvf *nic = nicvf_pmd_priv(dev);
int ret;
- ret = nicvf_start_rx_queue(dev, qidx);
+ if (qidx >= MAX_RCV_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_RCV_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_RCV_QUEUES_PER_QS;
+
+ ret = nicvf_vf_start_rx_queue(dev, nic, qidx);
if (ret)
return ret;
@@ -1087,7 +1105,14 @@ nicvf_dev_rx_queue_stop(struct rte_eth_dev *dev,
uint16_t qidx)
static int
nicvf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t qidx)
{
- return nicvf_start_tx_queue(dev, qidx);
+ struct nicvf *nic = nicvf_pmd_priv(dev);
+
+ if (qidx >= MAX_SND_QUEUES_PER_QS)
+ nic = nic->snicvf[(qidx / MAX_SND_QUEUES_PER_QS - 1)];
+
+ qidx = qidx % MAX_SND_QUEUES_PER_QS;
+
+ return nicvf_vf_start_tx_queue(dev, nic, qidx);
}
static int
@@ -1267,25 +1292,25 @@ rbdr_rte_mempool_get(void *dev, void *opaque)
}
static int
-nicvf_dev_start(struct rte_eth_d
[dpdk-dev] [PATCH 11/13] net/thunderx: add secondary qset support in device configure
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 65 +
1 file changed, 59 insertions(+), 6 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index 8510675..fcdbebf 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -163,6 +163,7 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
{
struct nicvf *nic = nicvf_pmd_priv(dev);
uint32_t buffsz, frame_size = mtu + ETHER_HDR_LEN + ETHER_CRC_LEN;
+ size_t i;
PMD_INIT_FUNC_TRACE();
@@ -198,6 +199,10 @@ nicvf_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
/* Update max frame size */
dev->data->dev_conf.rxmode.max_rx_pkt_len = (uint32_t)frame_size;
nic->mtu = mtu;
+
+ for (i = 0; i < nic->sqs_count; i++)
+ nic->snicvf[i]->mtu = mtu;
+
return 0;
}
@@ -507,7 +512,8 @@ nicvf_qset_cq_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
const struct rte_memzone *rz;
uint32_t ring_size = CMP_QUEUE_SZ_MAX * sizeof(union cq_entry_t);
- rz = rte_eth_dma_zone_reserve(dev, "cq_ring", qidx, ring_size,
+ rz = rte_eth_dma_zone_reserve(dev, "cq_ring",
+ nicvf_netdev_qidx(nic, qidx), ring_size,
NICVF_CQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for cq hw ring");
@@ -530,8 +536,9 @@ nicvf_qset_sq_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
const struct rte_memzone *rz;
uint32_t ring_size = SND_QUEUE_SZ_MAX * sizeof(union sq_entry_t);
- rz = rte_eth_dma_zone_reserve(dev, "sq", qidx, ring_size,
- NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "sq",
+ nicvf_netdev_qidx(nic, qidx), ring_size,
+ NICVF_SQ_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed allocate mem for sq hw ring");
return -ENOMEM;
@@ -563,8 +570,9 @@ nicvf_qset_rbdr_alloc(struct rte_eth_dev *dev, struct nicvf
*nic,
}
ring_size = sizeof(struct rbdr_entry_t) * RBDR_QUEUE_SZ_MAX;
- rz = rte_eth_dma_zone_reserve(dev, "rbdr", 0, ring_size,
- NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
+ rz = rte_eth_dma_zone_reserve(dev, "rbdr",
+ nicvf_netdev_qidx(nic, 0), ring_size,
+ NICVF_RBDR_BASE_ALIGN_BYTES, nic->node);
if (rz == NULL) {
PMD_INIT_LOG(ERR, "Failed to allocate mem for rbdr desc ring");
return -ENOMEM;
@@ -1678,12 +1686,37 @@ nicvf_dev_close(struct rte_eth_dev *dev)
}
static int
+nicvf_request_sqs(struct nicvf *nic)
+{
+ size_t i;
+
+ assert_if_secondary(nic);
+ assert(nic->sqs_count > 0);
+ assert(nic->sqs_count <= MAX_SQS_PER_VF);
+
+ /* Set no of Rx/Tx queues in each of the SQsets */
+ for (i = 0; i < nic->sqs_count; i++) {
+ if (nicvf_svf_empty())
+ rte_panic("Cannot assign sufficient number of "
+ "secondary queues to primary VF%" PRIu8 "\n",
+ nic->vf_id);
+
+ nic->snicvf[i] = nicvf_svf_pop();
+ nic->snicvf[i]->sqs_id = i;
+ }
+
+ return nicvf_mbox_request_sqs(nic);
+}
+
+static int
nicvf_dev_configure(struct rte_eth_dev *dev)
{
- struct rte_eth_conf *conf = &dev->data->dev_conf;
+ struct rte_eth_dev_data *data = dev->data;
+ struct rte_eth_conf *conf = &data->dev_conf;
struct rte_eth_rxmode *rxmode = &conf->rxmode;
struct rte_eth_txmode *txmode = &conf->txmode;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint8_t cqcount;
PMD_INIT_FUNC_TRACE();
@@ -1748,6 +1781,26 @@ nicvf_dev_configure(struct rte_eth_dev *dev)
return -EINVAL;
}
+ assert_if_secondary(nic);
+ NICVF_STATIC_ASSERT(MAX_RCV_QUEUES_PER_QS == MAX_SND_QUEUES_PER_QS);
+ cqcount = RTE_MAX(data->nb_tx_queues, data->nb_rx_queues);
+ if (cqcount > MAX_RCV_QUEUES_PER_QS) {
+ nic->sqs_count = RTE_ALIGN_CEIL(cqcount, MAX_RCV_QUEUES_PER_QS);
+ nic->sqs_count = (nic->sqs_count / MAX_RCV_QUEUES_PER_QS) - 1;
+ } else {
+
[dpdk-dev] [PATCH 12/13] net/thunderx: add final bits for secondary queue support
From: Kamil Rytarowski
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
drivers/net/thunderx/nicvf_ethdev.c | 178 +---
1 file changed, 144 insertions(+), 34 deletions(-)
diff --git a/drivers/net/thunderx/nicvf_ethdev.c
b/drivers/net/thunderx/nicvf_ethdev.c
index fcdbebf..04bfde0 100644
--- a/drivers/net/thunderx/nicvf_ethdev.c
+++ b/drivers/net/thunderx/nicvf_ethdev.c
@@ -118,7 +118,7 @@ nicvf_interrupt(void *arg)
nicvf_interrupt, dev);
}
-static void __rte_unused
+static void
nicvf_vf_interrupt(void *arg)
{
struct nicvf *nic = arg;
@@ -236,9 +236,15 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
struct nicvf_hw_tx_qstats tx_qstats;
struct nicvf_hw_stats port_stats;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint16_t rx_start, rx_end;
+ uint16_t tx_start, tx_end;
+ size_t i;
+
+ /* RX queue indices for the first VF */
+ nicvf_rx_range(dev, nic, &rx_start, &rx_end);
/* Reading per RX ring stats */
- for (qidx = 0; qidx < dev->data->nb_rx_queues; qidx++) {
+ for (qidx = rx_start; qidx <= rx_end; qidx++) {
if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
break;
@@ -247,8 +253,11 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
}
+ /* TX queue indices for the first VF */
+ nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+
/* Reading per TX ring stats */
- for (qidx = 0; qidx < dev->data->nb_tx_queues; qidx++) {
+ for (qidx = tx_start; qidx <= tx_end; qidx++) {
if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
break;
@@ -257,6 +266,40 @@ nicvf_dev_stats_get(struct rte_eth_dev *dev, struct
rte_eth_stats *stats)
stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
}
+ for (i = 0; i < nic->sqs_count; i++) {
+ struct nicvf *snic = nic->snicvf[i];
+
+ if (snic == NULL)
+ break;
+
+ /* RX queue indices for a secondary VF */
+ nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+
+ /* Reading per RX ring stats */
+ for (qidx = rx_start; qidx <= rx_end; qidx++) {
+ if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+ break;
+
+ nicvf_hw_get_rx_qstats(snic, &rx_qstats,
+ qidx % MAX_RCV_QUEUES_PER_QS);
+ stats->q_ibytes[qidx] = rx_qstats.q_rx_bytes;
+ stats->q_ipackets[qidx] = rx_qstats.q_rx_packets;
+ }
+
+ /* TX queue indices for a secondary VF */
+ nicvf_tx_range(dev, snic, &tx_start, &tx_end);
+ /* Reading per TX ring stats */
+ for (qidx = tx_start; qidx <= tx_end; qidx++) {
+ if (qidx == RTE_ETHDEV_QUEUE_STAT_CNTRS)
+ break;
+
+ nicvf_hw_get_tx_qstats(snic, &tx_qstats,
+ qidx % MAX_SND_QUEUES_PER_QS);
+ stats->q_obytes[qidx] = tx_qstats.q_tx_bytes;
+ stats->q_opackets[qidx] = tx_qstats.q_tx_packets;
+ }
+ }
+
nicvf_hw_get_stats(nic, &port_stats);
stats->ibytes = port_stats.rx_bytes;
stats->ipackets = port_stats.rx_ucast_frames;
@@ -322,13 +365,36 @@ nicvf_dev_stats_reset(struct rte_eth_dev *dev)
int i;
uint16_t rxqs = 0, txqs = 0;
struct nicvf *nic = nicvf_pmd_priv(dev);
+ uint16_t rx_start, rx_end;
+ uint16_t tx_start, tx_end;
- for (i = 0; i < dev->data->nb_rx_queues; i++)
+ /* Reset all primary nic counters */
+ nicvf_rx_range(dev, nic, &rx_start, &rx_end);
+ for (i = rx_start; i <= rx_end; i++)
rxqs |= (0x3 << (i * 2));
- for (i = 0; i < dev->data->nb_tx_queues; i++)
+
+ nicvf_tx_range(dev, nic, &tx_start, &tx_end);
+ for (i = tx_start; i <= tx_end; i++)
txqs |= (0x3 << (i * 2));
nicvf_mbox_reset_stat_counters(nic, 0x3FFF, 0x1F, rxqs, txqs);
+
+ /* Reset secondary nic queue counters */
+ for (i = 0; i < nic->sqs_count; i++) {
+ struct nicvf *snic = nic->snicvf[i];
+ if (snic == NULL)
+ break;
+
+ nicvf_rx_range(dev, snic, &rx_start, &rx_end);
+ for (i = rx_start; i
[dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue set support
From: Kamil Rytarowski Signed-off-by: Maciej Czekaj Signed-off-by: Kamil Rytarowski Signed-off-by: Zyta Szpak Signed-off-by: Slawomir Rosek Signed-off-by: Radoslaw Biernacki Signed-off-by: Jerin Jacob --- doc/guides/nics/thunderx.rst | 114 --- 1 file changed, 107 insertions(+), 7 deletions(-) diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index acc05a4..2b6f1f0 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -56,6 +56,7 @@ Features of the ThunderX PMD are: - VLAN stripping - SR-IOV VF - NUMA support +- Multi queue set support (up to 96 queues (12 queue sets)) per port Supported ThunderX SoCs --- @@ -322,6 +323,112 @@ This section provides instructions to configure SR-IOV with Linux OS. #. Refer to section :ref:`Running testpmd ` for instruction how to launch ``testpmd`` application. +Multiple Queue Set per DPDK port configuration +~~ + +There are two types of VFs: + +- Primary VF +- Secondary VF + +Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port. +In case port is configured to use more than 8 queues, then it requires one (or more) +secondary VF. Each secondary VF adds additional 8 queues to the queue set. + +During PMD driver initialization, the primary VF's are enumerated by checking the +specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set). +They are at the beginning of VF list (the remain ones are secondary VF's). + +The primary VFs are used as master queue sets. Secondary VFs provid +additional queue sets for primary ones. If a port is configured for more then +8 queues than it will request for additional queues from secondary VFs. + +Secondary VFs cannot be shared between primary VFs. + +Primary VFs are present on the beginning of the 'Network devices using kernel +driver' list, secondary VFs are on the remaining on the remaining part of the list. + + .. note:: + + The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`. + We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility. + + .. note:: + + Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to + this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs. + + +Example device binding +~~ + +If a system has three interfaces, a total of 18 VF devices will be created +on a non-NUMA machine. + + .. note:: + + NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port. + + .. code-block:: console + + # tools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + + + + Network devices using kernel driver + === + :01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + :01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic + 0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic + 0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic + 0002:01:02.0 'Device 0011' if= drv=thunder-nicvf
[dpdk-dev] [PATCH 1/2] ethdev: Allow to overload pci_drv.devinit and pci_drv.devuninit
W dniu 03.02.2016 o 09:47, David Marchand pisze:
> Hello,
Hello,
> On Tue, Feb 2, 2016 at 3:27 PM, wrote:
>> From: Kamil Rytarowski
>>
>> This change enables drivers needing custom pci (de)initialization functions
>> through the standard callback overloading.
>>
>> For example:
>>
>> /*
>>* virtual function driver struct
>>*/
>> static struct eth_driver rte_DRIVER_pmd = {
>> {
>> .name = "rte_DRIVER_pmd",
>> .id_table = pci_id_DRIVER_map,
>> .drv_flags = RTE_PCI_DRV_NEED_MAPPING,
>> .devinit = DRIVER_pci_devinit,
>> },
>> .eth_dev_init = eth_DRIVER_dev_init,
>> .dev_private_size = sizeof(struct DRIVER),
>> };
>>
>> Use-case is as follows: NIC offers several pci virtual functions, while
>> one of them is to be treated as port, we need to configure the rest in a
>> specific way for particular device for full interface (port) functionality.
> Mmm, why don't you register a custom pci driver rather than a eth_driver ?
Both Virtual Functions are of eth_driver type and they share the same
PCI ID.
They are like 2 (or more) NIC devices working in tandem (it's not
bonding), with a possibility to join into a single one and offer
additional functionality. In that case one is master and the other is
donor of functionality, like additional queue sets.
> And do your custom things in its devinit function ?
>
>
Yes, I'm doing custom things.
I'm requesting from PF the mode of the device to be initialized. This
part is handled dynamically and depends of the current configuration in PF.
In my use-case there are two device types: primary (master) and
secondary (slave). For the primary VF I'm creating a DPDK port normally,
for secondary I retain configured PCI device for further reuse (and
there is no port created).
I cannot overload functions called after .devinit() as the secondary
devices will be wrongly treated like a DPDK port.
My use-case doesn't need modified API to be functional.
The missing handling of overloaded .devinit and .devuninit looks like a
bug - there is API for it, but it will keep overwriting the pointers
with local functions (rte_eth_dev_init(), rte_eth_dev_uninit()).
[dpdk-dev] [PATCH 1/2] ethdev: Allow to overload pci_drv.devinit and pci_drv.devuninit
W dniu 03.02.2016 o 15:08, David Marchand pisze: > On Wed, Feb 3, 2016 at 12:39 PM, Kamil Rytarowski > wrote: >> W dniu 03.02.2016 o 09:47, David Marchand pisze: >>> And do your custom things in its devinit function ? >> I'm requesting from PF the mode of the device to be initialized. This part >> is handled dynamically and depends of the current configuration in PF. >> >> In my use-case there are two device types: primary (master) and secondary >> (slave). For the primary VF I'm creating a DPDK port normally, for secondary >> I retain configured PCI device for further reuse (and there is no port >> created). > Well, again, if you don't want to associate a port to this pci > resource, why are you registering a eth_driver ? > A eth_driver driver supposes you have a 1 - 1 relation between ethdev > and pci resource. In my use-case one DPDK port optionally manages more than single PCI resource, and these PCI resources compose single interface. Another example of overloaded .devinit is in app/test/test_pci.c: /* * PCI test * * * - Register a driver with a ``devinit()`` function. * * - Dump all PCI devices. * * - Check that the ``devinit()`` function is called at least once. */ With the current implementation it won't work, as .devinit callback will be overwritten by the internal function. > For your case, register a pci driver, then in your pci probing > function (.devinit), depending on what you want to do, you can either > do nothing (?) or create one or more ethdevs (see mlx* and cxgbe > drivers). > > > This is what I'm doing right now. I need to initialize PCI bars and interrupts (resources) - all having the same PCI ID and their functionality depending upon PF configuration. Depending on this state, I'm making further decisions in DRIVER_devinit() and whether to make from it a port or a resource to reuse by a master port.
[dpdk-dev] [PATCH 1/2] ethdev: Allow to overload pci_drv.devinit and pci_drv.devuninit
W dniu 03.02.2016 o 16:49, Kamil Rytarowski pisze: > > > W dniu 03.02.2016 o 15:08, David Marchand pisze: >> On Wed, Feb 3, 2016 at 12:39 PM, Kamil Rytarowski >> wrote: >>> W dniu 03.02.2016 o 09:47, David Marchand pisze: >>>> And do your custom things in its devinit function ? >>> I'm requesting from PF the mode of the device to be initialized. >>> This part >>> is handled dynamically and depends of the current configuration in PF. >>> >>> In my use-case there are two device types: primary (master) and >>> secondary >>> (slave). For the primary VF I'm creating a DPDK port normally, for >>> secondary >>> I retain configured PCI device for further reuse (and there is no port >>> created). >> Well, again, if you don't want to associate a port to this pci >> resource, why are you registering a eth_driver ? >> A eth_driver driver supposes you have a 1 - 1 relation between ethdev >> and pci resource. > In my use-case one DPDK port optionally manages more than single PCI > resource, and these PCI resources compose single interface. > > Another example of overloaded .devinit is in app/test/test_pci.c: > > /* > * PCI test > * > * > * - Register a driver with a ``devinit()`` function. > * > * - Dump all PCI devices. > * > * - Check that the ``devinit()`` function is called at least once. > */ > > With the current implementation it won't work, as .devinit callback > will be overwritten by the internal function. > >> For your case, register a pci driver, then in your pci probing >> function (.devinit), depending on what you want to do, you can either >> do nothing (?) or create one or more ethdevs (see mlx* and cxgbe >> drivers). >> >> >> > > This is what I'm doing right now. > > I need to initialize PCI bars and interrupts (resources) - all having > the same PCI ID and their functionality depending upon PF > configuration. Depending on this state, I'm making further decisions > in DRIVER_devinit() and whether to make from it a port or a resource > to reuse by a master port. After sending the mail I noted that I can work with the .init callback and I may be wrong. I will have a closer look and I will check whether it will work for me. Thank you,
[dpdk-dev] [PATCH v7 1/2] tools: Add support for handling built-in kernel modules
ping? W dniu 29.01.2016 o 08:21, Yuanhan Liu pisze: > On Thu, Jan 28, 2016 at 02:13:53PM +0100, krytarowski at caviumnetworks.com > wrote: >> From: Kamil Rytarowski >> >> Currently dpdk_nic_bind.py detects Linux kernel modules via reading >> /proc/modules. Built-in ones aren't listed there and therefore they are not >> being found by the script. >> >> Add support for checking built-in modules with parsing the sysfs files. >> >> This commit obsoletes the /proc/modules parsing approach. >> >> Signed-off-by: Kamil Rytarowski > Acked-by: Yuanhan Liu > > --yliu
[dpdk-dev] [PATCH v7 2/2] eal/linux: Add support for handling built-in kernel modules
Thank you! W dniu 09.02.2016 o 15:56, Thomas Monjalon pisze: > 2016-01-28 14:13, krytarowski at caviumnetworks.com: >> From: Kamil Rytarowski >> >> Currently rte_eal_check_module() detects Linux kernel modules via reading >> /proc/modules. Built-in ones aren't listed there and therefore they are not >> being found by the script. >> >> Add support for checking built-in modules with parsing the sysfs files >> >> This commit obsoletes the /proc/modules parsing approach. >> >> Signed-off-by: Kamil Rytarowski >> Acked-by: David Marchand >> Acked-by: Yuanhan Liu > An include is missing: > #include > After adding this line, > Series applied, thanks >
[dpdk-dev] [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains
W dniu 10.10.2016 o 12:19, Ferruh Yigit pisze:
> Hi Kamil,
>
> On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
>> It's possible to have the same numbers for bus, device id and function,
>> therefore we need to differentiate on domain.
>>
>> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
>>
>> Signed-off-by: Maciej Czekaj
>> Signed-off-by: Kamil Rytarowski
>> Signed-off-by: Zyta Szpak
>> Signed-off-by: Slawomir Rosek
>> Signed-off-by: Radoslaw Biernacki
>> Signed-off-by: Jerin Jacob
>> ---
>> lib/librte_ether/rte_ethdev.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>> index 382c959..01d5fb0 100644
>> --- a/lib/librte_ether/rte_ethdev.c
>> +++ b/lib/librte_ether/rte_ethdev.c
>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t
>> size,
>> {
>> int ret;
>>
>> -ret = snprintf(name, size, "%d:%d.%d",
>> +ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>> pci_dev->addr.bus, pci_dev->addr.devid,
>> pci_dev->addr.function);
>> if (ret < 0)
>>
> Is it possible to separate this patch from patchset, this is a ethdev
> patch and it seems not directly related to the rest of the patchset?
>
> Thanks,
> ferruh
This patch is directly related with secondary queue set support on
ThunderX, but it can be skipped in this chain of patches and applied as
a standalone diff.
Is disabling this one on patch work sufficient? Of course unless there
are no more comments to produce v3 of the original patch chain "Add
support for secondary queue set in nicvf thunderx driver".
Should I resubmit it as a new standalone patch?
[dpdk-dev] [PATCH] ethdev: Support VFs on the different PCI domains
It's possible to have the same numbers for bus, device id and function,
therefore we need to differentiate on domain.
This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
Signed-off-by: Maciej Czekaj
Signed-off-by: Kamil Rytarowski
Signed-off-by: Zyta Szpak
Signed-off-by: Slawomir Rosek
Signed-off-by: Radoslaw Biernacki
Signed-off-by: Jerin Jacob
---
lib/librte_ether/rte_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 382c959..01d5fb0 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t
size,
{
int ret;
- ret = snprintf(name, size, "%d:%d.%d",
+ ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
pci_dev->addr.bus, pci_dev->addr.devid,
pci_dev->addr.function);
if (ret < 0)
--
1.9.1
[dpdk-dev] [PATCH v2 14/15] ethdev: Support VFs on the different PCI domains
W dniu 10.10.2016 o 15:27, Ferruh Yigit pisze:
> On 10/10/2016 2:01 PM, Kamil Rytarowski wrote:
>>
>> W dniu 10.10.2016 o 12:19, Ferruh Yigit pisze:
>>> Hi Kamil,
>>>
>>> On 9/30/2016 1:05 PM, Kamil Rytarowski wrote:
>>>> It's possible to have the same numbers for bus, device id and function,
>>>> therefore we need to differentiate on domain.
>>>>
>>>> This enables DPDK with multiple VFs on ThunderX 2-socket hardware.
>>>>
>>>> Signed-off-by: Maciej Czekaj
>>>> Signed-off-by: Kamil Rytarowski
>>>> Signed-off-by: Zyta Szpak
>>>> Signed-off-by: Slawomir Rosek
>>>> Signed-off-by: Radoslaw Biernacki
>>>> Signed-off-by: Jerin Jacob
>>>> ---
>>>>lib/librte_ether/rte_ethdev.c | 2 +-
>>>>1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>>>> index 382c959..01d5fb0 100644
>>>> --- a/lib/librte_ether/rte_ethdev.c
>>>> +++ b/lib/librte_ether/rte_ethdev.c
>>>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name,
>>>> size_t size,
>>>>{
>>>>int ret;
>>>>
>>>> - ret = snprintf(name, size, "%d:%d.%d",
>>>> + ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>>>>pci_dev->addr.bus, pci_dev->addr.devid,
>>>>pci_dev->addr.function);
>>>>if (ret < 0)
>>>>
>>> Is it possible to separate this patch from patchset, this is a ethdev
>>> patch and it seems not directly related to the rest of the patchset?
>>>
>>> Thanks,
>>> ferruh
>> This patch is directly related with secondary queue set support on
>> ThunderX, but it can be skipped in this chain of patches and applied as
>> a standalone diff.
>>
>> Is disabling this one on patch work sufficient? Of course unless there
>> are no more comments to produce v3 of the original patch chain "Add
>> support for secondary queue set in nicvf thunderx driver".
> I think it is sufficient, at least I don't have any more comment for
> rest of the patchset and it looks good to me.
>
>> Should I resubmit it as a new standalone patch?
> Can you please resubmit just this one patch, so it can be properly reviewed.
>
> Thanks,
> ferruh
>
Hi,
I've performed the needed actions.
This patch should be disabled in the patch-chain and has been resent as
a new one.
[dpdk-dev] [PATCH] ethdev: Support VFs on the different PCI domains
W dniu 11.10.2016 o 18:52, David Marchand pisze:
> On Tue, Oct 11, 2016 at 3:49 PM, Kamil Rytarowski
> wrote:
>> diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
>> index 382c959..01d5fb0 100644
>> --- a/lib/librte_ether/rte_ethdev.c
>> +++ b/lib/librte_ether/rte_ethdev.c
>> @@ -225,7 +225,7 @@ rte_eth_dev_create_unique_device_name(char *name, size_t
>> size,
>> {
>> int ret;
>>
>> - ret = snprintf(name, size, "%d:%d.%d",
>> + ret = snprintf(name, size, "%d:%d:%d.%d", pci_dev->addr.domain,
>> pci_dev->addr.bus, pci_dev->addr.devid,
>> pci_dev->addr.function);
>> if (ret < 0)
> This patch is obsolete since this part has been moved to eal.
>
> Can you test with current master branch if there is still an issue ?
> Thanks.
>
>
I've tested DPDK master without this PCI domain patch and everything
seems to work.
Thank you.
