On Mon, May 16, 2016 at 7:13 PM, Stokes, Ian <ian.sto...@intel.com> wrote:
> > -----Original Message----- > > From: Mauricio Vasquez B > > [mailto:mauricio.vasquezber...@studenti.polito.it] > > Sent: Friday, May 13, 2016 9:26 PM > > To: dev@openvswitch.org > > Cc: acon...@redhat.com; l...@cn.fujitsu.com; mchand...@suse.de; Stokes, > > Ian; f...@sysclose.org; Traynor, Kevin; diproiet...@vmware.com > > Subject: [PATCH v5] netdev-dpdk: add hotplug support > > > > In order to use dpdk ports in ovs they have to be bound to a DPDK > > compatible driver before ovs is started. > > > > This patch adds the possibility to hotplug (or hot-unplug) a device > > after ovs has been started. The implementation adds two appctl commands: > > netdev-dpdk/port-attach and netdev-dpdk/port-detach > > > > After the user attaches a new device, it has to be added to a bridge > > using the add-port command, similarly, before detaching a device, it has > > to be removed using the del-port command. > > > > Signed-off-by: Mauricio Vasquez B > > <mauricio.vasquezber...@studenti.polito.it> > > --- > > v5: > > - use two appctl commands instead of a single one > > - rebase to master > > v4: > > - fix typo in commit message > > - remove unnecessary whitespace change in INSTALL.DPDK.md > > v3: > > - create dpdk_port_attach and dpdk_port_detach functions > > - modify mutex locking order > > v2: > > - use rte_eth_dev_is_valid_port() to check if a port is valid > > INSTALL.DPDK.md | 24 +++++++++++++ > > NEWS | 1 + > > lib/netdev-dpdk.c | 101 > > ++++++++++++++++++++++++++++++++++++++++++++++++++---- > > 3 files changed, 119 insertions(+), 7 deletions(-) > > > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 93f92e4..8d5a0e0 > > 100644 > > --- a/INSTALL.DPDK.md > > +++ b/INSTALL.DPDK.md > > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd: > > For more details regarding egress-policer parameters please refer to > > the > > vswitch.xml. > > > > +9. Port Hotplug > > + > > + Ovs supports port hotplugging, it allows to use ports that were not > > bound > > + to DPDK when vswitchd was started. > > + In order to attach a port, it has to be bound to DPDK using the > > + dpdk_nic_bind.py script: > > + > > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` > > + > > + Then it can be attached to OVS: > > + > > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` > > + > > + At this point, the user can create a ovs port using the add-port > > command. > > + > > + It is also possible to detach a port from ovs, the user has to > > remove the > > + port using the del-port command, then it can be detached using: > > + > > + `ovs-appctl netdev-dpdk/port-detach dpdk0` > > + > > + This feature is not supported by all the NICs, please refer to the > > + [DPDK Port Hotplug Framework] in order to get more information. > > + > > Performance Tuning: > > ------------------- > > > > @@ -999,3 +1022,4 @@ Please report problems to b...@openvswitch.org. > > [INSTALL.md]:INSTALL.md > > [DPDK Linux GSG]: > > http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and- > > unbinding-network-ports-to-from-the-igb-uioor-vfio-modules > > [DPDK Docs]: http://dpdk.org/doc > > +[DPDK Port Hotplug Framework]: > > +http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html > > diff --git a/NEWS b/NEWS > > index 4e81cad..d89d9a7 100644 > > --- a/NEWS > > +++ b/NEWS > > @@ -32,6 +32,7 @@ Post-v2.5.0 > > * DB entries have been added for many of the DPDK EAL command line > > arguments. Additional arguments can be passed via the dpdk-extra > > entry. > > + * Port Hotplug is now supported. > > - ovs-benchmark: This utility has been removed due to lack of use > > and > > bitrot. > > - ovs-appctl: > > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index > > af86d19..6159a60 100644 > > --- a/lib/netdev-dpdk.c > > +++ b/lib/netdev-dpdk.c > > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) > > OVS_REQUIRES(dpdk_mutex) > > int diag; > > int n_rxq, n_txq; > > > > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { > > + if (!rte_eth_dev_is_valid_port(dev->port_id)) { > > return ENODEV; > > } > > > > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn > > *conn, int argc, > > unixctl_command_reply(conn, "OK"); > > } > > > > +static void > > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) { > > + int ret; > > + char response[128]; > > + uint8_t port_id; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = rte_eth_dev_attach(argv[1], &port_id); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Error attaching device '%s'", argv[1]); > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); > > + return; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Device '%s' has been attached as 'dpdk%d'", argv[1], > > + port_id); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); } > > + > > +static void > > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, > > + const char *argv[], void *aux OVS_UNUSED) { > > + int ret; > > + char response[128]; > > + unsigned int parsed_port; > > + uint8_t port_id; > > + char devname[RTE_ETH_NAME_MAX_LEN]; > > + > > + ovs_mutex_lock(&dpdk_mutex); > > + > > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); > > + if (ret) { > > + snprintf(response, sizeof(response), > > + "'%s' is not a valid port", argv[1]); > > + goto error; > > + } > > + > > + port_id = parsed_port; > > + > > + struct netdev *netdev = netdev_from_name(argv[1]); > > + if (netdev) { > > + netdev_close(netdev); > > + snprintf(response, sizeof(response), > > + "Port '%s' is being used. Remove it before detaching", > > + argv[1]); > > + goto error; > > + } > > + > > + rte_eth_dev_close(port_id); > > + > > + ret = rte_eth_dev_detach(port_id, devname); > > + if (ret < 0) { > > + snprintf(response, sizeof(response), > > + "Port '%s' can not be detached", argv[1]); > > + goto error; > > + } > > + > > + snprintf(response, sizeof(response), > > + "Port '%s' has been detached", argv[1]); > > + > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply(conn, response); > > + return; > > + > > +error: > > + ovs_mutex_unlock(&dpdk_mutex); > > + unixctl_command_reply_error(conn, response); } > > + > > /* > > * Set virtqueue flags so that we do not receive interrupts. > > */ > > @@ -2404,6 +2481,15 @@ dpdk_common_init(void) > > "[netdev] up|down", 1, 2, > > netdev_dpdk_set_admin_state, NULL); > > > > + unixctl_command_register("netdev-dpdk/port-attach", > > + "pci address of device", 1, 1, > > + netdev_dpdk_port_attach, NULL); > > + > > + unixctl_command_register("netdev-dpdk/port-detach", > > + "port", 1, 1, > > + netdev_dpdk_port_detach, NULL); > > + > > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); > > } > > > > /* Client Rings */ > > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned > > int port_no, { > > struct dpdk_ring *ivshmem; > > char ring_name[RTE_RING_NAMESIZE]; > > - int err; > > + int err, port_id; > > > > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); > > if (ivshmem == NULL) { > > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned > > int port_no, > > return ENOMEM; > > } > > > > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > - &ivshmem->cring_tx, 1, SOCKET0); > > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, > > + &ivshmem->cring_tx, 1, SOCKET0); > > > > - if (err < 0) { > > + if (port_id < 0) { > > rte_free(ivshmem); > > return ENODEV; > > } > > > > ivshmem->user_port_id = port_no; > > - ivshmem->eth_port_id = rte_eth_dev_count() - 1; > > + ivshmem->eth_port_id = port_id; > > + *eth_port_id = port_id; > > + > > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); > > > > - *eth_port_id = ivshmem->eth_port_id; > > return 0; > > } > > > > -- > > 1.9.1 > > Thanks for the latest patch revision Mauricio. > I've tested it with an Intel 82599 NIC and it works without issue. > > However I did come across an issue related to the XL710 interface when it > is attached, detached and then re-attached, essentially the rte_init > function fails as the device appears as busy. I think this issue is > specific to the Xl710 however as the same behavior could not be replicated > with the Intel 82599 NIC. > > As you have it called out in your notes that hot plug support is interface > dependent I guess this is ok. It's seems to be a DPDK issue for the XL710 > and will need further investigation separate to this work. > > Unfortunately I don't have any XL170 to test it here, if you want, you can use the testpmd DPDK application to test the hot-plug support and discuss results with the DPDK developers. > Tested-by: ian.sto...@intel.com > > Thanks > Ian > Thanks for testing! _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
