This patch adds support for a new port type to userspace datapath
called dpdkvhost. This allows KVM (QEMU) to offload the servicing
of virtio-net devices to its associated dpdkvhost port. Instructions
for use are in INSTALL.DPDK.
This has been tested on Intel multi-core platforms and with clients
that have virtio-net interfaces.
ver 8:
- rebased with master
ver 7:
- rebased with master
- reworked INSTALL.DPDK and performed extra testing based on
review comments
ver 6:
- rebased with master
- modified to use DPDK v1.8.0 vhost library
- reworked for review comments
ver 5:
- rebased against latest master
ver 4:
- added eventfd_link.h and eventfd_link.c to EXTRA_DIST in
utilities/automake.mk
- rebased with master to work with DPDK 1.7 ver 3:
- rebased with master
ver 2:
- rebased with master
Signed-off-by: Ciara Loftus <ciara.lof...@intel.com>
Signed-off-by: Kevin Traynor <kevin.tray...@intel.com>
Signed-off-by: Maryam Tahhan <maryam.tah...@intel.com>
---
INSTALL.DPDK.md | 260 +++++++++++++++++++
Makefile.am | 4 +
lib/automake.mk | 1 +
lib/netdev-dpdk.c | 649 +++++++++++++++++++++++++++++++++++++++--------
lib/netdev.c | 3 +-
utilities/automake.mk | 3 +-
utilities/qemu-wrap.py | 389 ++++++++++++++++++++++++++++
vswitchd/ovs-vswitchd.c | 4 +-
8 files changed, 1201 insertions(+), 112 deletions(-)
create mode 100755 utilities/qemu-wrap.py
diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md
index 276fe56..4deeb87 100644
--- a/INSTALL.DPDK.md
+++ b/INSTALL.DPDK.md
@@ -17,6 +17,7 @@ Building and Installing:
------------------------
Required DPDK 1.8.0
+Optional `fuse`, `fuse-devel`
1. Configure build & install DPDK:
1. Set `$DPDK_DIR`
@@ -290,6 +291,264 @@ A general rule of thumb for better performance is that
the client
application should not be assigned the same dpdk core mask "-c" as
the vswitchd.
+DPDK vhost:
+-----------
+
+vhost-cuse is only supported at present i.e. not using the standard QEMU
+vhost-user interface. It is intended that vhost-user support will be added
+in future releases when supported in DPDK and that vhost-cuse will eventually
+be deprecated. See [DPDK Docs] for more info on vhost.
+
+Prerequisites:
+1. DPDK 1.8 with vhost support enabled and recompile OVS as above.
+
+ Update `config/common_linuxapp` so that DPDK is built with vhost
+ libraries:
+
+ `CONFIG_RTE_LIBRTE_VHOST=y`
+
+2. Insert the Cuse module:
+
+ `modprobe cuse`
+
+3. Build and insert the `eventfd_link` module:
+
+ `cd $DPDK_DIR/lib/librte_vhost/eventfd_link/`
+ `make`
+ `insmod $DPDK_DIR/lib/librte_vhost/eventfd_link.ko`
+
+Following the steps above to create a bridge, you can now add DPDK vhost
+as a port to the vswitch.
+
+`ovs-vsctl add-port br0 dpdkvhost0 -- set Interface dpdkvhost0 type=dpdkvhost`
+
+Unlike DPDK ring ports, DPDK vhost ports can have arbitrary names:
+
+`ovs-vsctl add-port br0 port123ABC -- set Interface port123ABC type=dpdkvhost`
+
+However, please note that when attaching userspace devices to QEMU, the
+name provided during the add-port operation must match the ifname parameter
+on the QEMU command line.
+
+
+DPDK vhost VM configuration:
+----------------------------
+
+ vhost ports use a Linux* character device to communicate with QEMU.
+ By default it is set to `/dev/vhost-net`. It is possible to reuse this
+ standard device for DPDK vhost, which makes setup a little simpler but it
+ is better practice to specify an alternative character device in order to
+ avoid any conflicts if kernel vhost is to be used in parallel.
+
+1. This step is only needed if using an alternative character device.
+
+ The new character device filename must be specified on the vswitchd
+ commandline:
+
+ `./vswitchd/ovs-vswitchd --dpdk --basename my-vhost-net -c 0x1 ...`
+
+ Note that the `--basename` argument and associated string must be the first
+ arguments after `--dpdk` and come before the EAL arguments. In the example
+ above, the character device to be used will be `/dev/my-vhost-net`.
+
+2. This step is only needed if reusing the standard character device. It will
+ conflict with the kernel vhost character device so the user must first
+ remove it.
+
+ `rm -rf /dev/vhost-net`
+
+3a. Configure virtio-net adaptors:
+ The following parameters must be passed to the QEMU binary:
+
+ ```
+ -netdev tap,id=<id>,script=no,downscript=no,ifname=<name>,vhost=on
+ -device virtio-net-pci,netdev=net1,mac=<mac>
+ ```
+
+ Repeat the above parameters for multiple devices.
+
+ The DPDK vhost library will negiotiate its own features, so they
+ need not be passed in as command line params. Note that as offloads are
+ disabled this is the equivalent of setting:
+
+ `csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off`
+
+3b. If using an alternative character device. It must be also explicitly
+ passed to QEMU using the `vhostfd` argument:
+
+ ```
+ -netdev tap,id=<id>,script=no,downscript=no,ifname=<name>,vhost=on,
+ vhostfd=<open_fd>
+ -device virtio-net-pci,netdev=net1,mac=<mac>
+ ```
+
+ The open file descriptor must be passed to QEMU running as a child
+ process. This could be done with a simple python script.
+
+ ```
+ #!/usr/bin/python
+ fd = os.open("/dev/usvhost-1", os.O_RDWR)
+ subprocess.call("qemu-system-x86_64 .... -netdev tap,id=vhostnet0,\
+ vhost=on,vhostfd=" + fd +"...", shell=True)
+
+ Alternatively the the `qemu-wrap.py` script can be used to automate the
+ requirements specified above and can be used in conjunction with libvirt if
+ desired. See the "DPDK vhost VM configuration with QEMU wrapper" section
+ below.
+
+4. Configure huge pages:
+ QEMU must allocate the VM's memory on hugetlbfs. Vhost ports access a
+ virtio-net device's virtual rings and packet buffers mapping the VM's
+ physical memory on hugetlbfs. To enable vhost-ports to map the VM's
+ memory into their process address space, pass the following paramters
+ to QEMU:
+
+ `-object memory-backend-file,id=mem,size=4096,mem-path=/dev/hugepages,
+ share=on -numa node,memdev=mem -mem-prealloc`
+
+
+Note, if using the DPDK test-pmd sample application in the guest, then DPDK in
+the guest must be built without any modifications to the default
+`config/common_linuxapp` file. e.g. with
+
+ `CONFIG_RTE_BUILD_COMBINE_LIBS=n` and
+ `CONFIG_RTE_LIBRTE_VHOST=n`
+
+
+DPDK vhost VM configuration with QEMU wrapper:
+----------------------------------------------
+
+The QEMU wrapper script automatically detects and calls QEMU with the
+necessary parameters. It performs the following actions:
+
+ * Automatically detects the location of the hugetlbfs and inserts this
+ into the command line parameters.
+ * Automatically open file descriptors for each virtio-net device and
+ inserts this into the command line parameters.
+ * Calls QEMU passing both the command line parameters passed to the
+ script itself and those it has auto-detected.
+
+Before use, you **must** edit the configuration parameters section of the
+script to point to the correct emulator location and set additional
+settings. Of these settings, `emul_path` and `us_vhost_path` **must** be
+set. All other settings are optional.
+
+To use directly from the command line simply pass the wrapper some of the
+QEMU parameters: it will configure the rest. For example:
+
+```
+qemu-wrap.py -cpu host -boot c -hda <disk image> -m 4096 -smp 4
+ --enable-kvm -nographic -vnc none -net none -netdev tap,id=net1,
+ script=no,downscript=no,ifname=if1,vhost=on -device virtio-net-pci,
+ netdev=net1,mac=00:00:00:00:00:01
+
+DPDK vhost VM configuration with libvirt:
+-----------------------------------------
+
+If you are using libvirt, you must enable libvirt to access the character
+device by adding it to controllers cgroup for libvirtd using the following
+steps.
+
+ 1. In `/etc/libvirt/qemu.conf` add/edit the following lines:
+
+ ```
+ 1) clear_emulator_capabilities = 0
+ 2) user = "root"
+ 3) group = "root"
+ 4) cgroup_device_acl = [
+ "/dev/null", "/dev/full", "/dev/zero",
+ "/dev/random", "/dev/urandom",
+ "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
+ "/dev/rtc", "/dev/hpet", "/dev/net/tun",
+ "/dev/<my-vhost-device>",
+ "/dev/hugepages"]
+ ```
+
+ <my-vhost-device> refers to "vhost-net" if using the `/dev/vhost-net`
+ device. If you have specificed a different name on the ovs-vswitchd
+ commandline using the "--basename" parameter, please specify that
+ filename instead.
+
+ 2. Disable SELinux or set to permissive mode
+
+ 3. Restart the libvirtd process
+ For example, on Fedora:
+
+ `systemctl restart libvirtd.service`
+
+After successfully editing the configuration, you may launch your
+vhost-enabled VM. The XML describing the VM can be configured like so
+within the <qemu:commandline> section:
+
+ 1. Set up shared hugepages:
+
+ ```
+ <qemu:arg value='-object'/>
+ <qemu:arg
value='memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on'/>
+ <qemu:arg value='-numa'/>
+ <qemu:arg value='node,memdev=mem'/>
+ <qemu:arg value='-mem-prealloc'/>
+ ```
+
+ 2. Set up your tap devices:
+
+ ```
+ <qemu:arg value='-netdev'/>
+ <qemu:arg
value='type=tap,id=net1,script=no,downscript=no,ifname=vhost0,vhost=on'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value='virtio-net-pci,netdev=net1,mac=00:00:00:00:00:01'/>
+ ```
+
+ Repeat for as many devices as are desired, modifying the id, ifname
+ and mac as necessary.
+
+ Again, if you are using an alternative character device (other than
+ `/dev/vhost-net`), please specify the file descriptor like so:
+
+ `<qemu:arg
value='type=tap,id=net3,script=no,downscript=no,ifname=vhost0,vhost=on,vhostfd=<open_fd>'/>`
+
+ Where <open_fd> refers to the open file descriptor of the character
device.
+ Instructions of how to retrieve the file descriptor can be found in the
+ "DPDK vhost VM configuration" section.
+ Alternatively, the process is automated with the qemu-wrap.py script,
+ detailed in the next section.
+
+Now you may launch your VM using virt-manager, or like so:
+
+ `virsh create my_vhost_vm.xml`
+
+DPDK vhost VM configuration with libvirt and QEMU wrapper:
+----------------------------------------------------------
+
+To use the qemu-wrapper script in conjuntion with libvirt, follow the
+steps in the previous section before proceeding with the following steps:
+
+ 1. Place `qemu-wrap.py` in libvirtd's binary search PATH ($PATH)
+ Ideally in the same directory that the QEMU binary is located.
+
+ 2. Ensure that the script has the same owner/group and file permissions
+ as the QEMU binary.
+
+ 3. Update the VM xml file using "virsh edit VM.xml"
+
+ 1. Set the VM to use the launch script.
+ Set the emulator path contained in the `<emulator><emulator/>` tags.
+ For example, replace:
+
+ `<emulator>/usr/bin/qemu-kvm<emulator/>`
+
+ with:
+
+ `<emulator>/usr/bin/qemu-wrap.py<emulator/>`
+
+ 4. Edit the Configuration Parameters section of the script to point to
+ the correct emulator location and set any additional options. If you are
+ using a alternative character device name, please set "us_vhost_path" to the
+ location of that device. The script will automatically detect and insert
+ the correct "vhostfd" value in the QEMU command line arguements.
+
+ 5. Use virt-manager to launch the VM
+
Restrictions:
-------------
@@ -311,3 +570,4 @@ Please report problems to b...@openvswitch.org.
[INSTALL.userspace.md]:INSTALL.userspace.md
[INSTALL.md]:INSTALL.md
[DPDK Linux GSG]:
http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules
+[DPDK Docs]: http://dpdk.org/doc
diff --git a/Makefile.am b/Makefile.am
index 0480d20..e1a4e17 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -32,6 +32,10 @@ AM_CFLAGS = -Wstrict-prototypes
AM_CFLAGS += $(WARNING_FLAGS)
AM_CFLAGS += $(OVS_CFLAGS)
+if DPDK_NETDEV
+AM_CFLAGS += -D_FILE_OFFSET_BITS=64
+endif
+
if NDEBUG
AM_CPPFLAGS += -DNDEBUG
AM_CFLAGS += -fomit-frame-pointer
diff --git a/lib/automake.mk b/lib/automake.mk
index 2acfe18..594dec4 100644
--- a/lib/automake.mk
+++ b/lib/automake.mk
@@ -346,6 +346,7 @@ lib_libopenvswitch_la_SOURCES += \
endif
if DPDK_NETDEV
+lib_libopenvswitch_la_LDFLAGS += -lfuse
lib_libopenvswitch_la_SOURCES += \
lib/netdev-dpdk.c \
lib/netdev-dpdk.h
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index 1ba8310..8714c52 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -49,6 +49,7 @@
#include "rte_config.h"
#include "rte_mbuf.h"
+#include "rte_virtio_net.h"
VLOG_DEFINE_THIS_MODULE(dpdk);
static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20);
@@ -58,6 +59,7 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5,
20);
#define OVS_CACHE_LINE_SIZE CACHE_LINE_SIZE
#define OVS_VPORT_DPDK "ovs_dpdk"
+
/*
* need to reserve tons of extra space in the mbufs so we can align the
* DMA addresses to 4KB.
@@ -84,6 +86,12 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5,
20);
#define TX_HTHRESH 0 /* Default values of TX host threshold reg. */
#define TX_WTHRESH 0 /* Default values of TX write-back threshold reg. */
+#define MAX_BASENAME_SZ NAME_MAX /* Maximum character device basename size. */
+#define MAX_PKT_BURST 32 /* Max burst size for RX/TX */
+
+/* Character device basename. */
+char *dev_basename = NULL;
+
static const struct rte_eth_conf port_conf = {
.rxmode = {
.mq_mode = ETH_MQ_RX_RSS,
@@ -130,29 +138,30 @@ enum { MAX_TX_QUEUE_LEN = 384 };
enum { DPDK_RING_SIZE = 256 };
BUILD_ASSERT_DECL(IS_POW2(DPDK_RING_SIZE));
enum { DRAIN_TSC = 200000ULL };
+enum { DPDK = 0, VHOST };
static int rte_eal_init_ret = ENODEV;
-static struct ovs_mutex dpdk_mutex = OVS_MUTEX_INITIALIZER;
+static rte_spinlock_t dpdk_spinlock = RTE_SPINLOCK_INITIALIZER;
/* Contains all 'struct dpdk_dev's. */
-static struct ovs_list dpdk_list OVS_GUARDED_BY(dpdk_mutex)
+static struct ovs_list dpdk_list OVS_GUARDED_BY(dpdk_spinlock)
= OVS_LIST_INITIALIZER(&dpdk_list);
-static struct ovs_list dpdk_mp_list OVS_GUARDED_BY(dpdk_mutex)
+static struct ovs_list dpdk_mp_list OVS_GUARDED_BY(dpdk_spinlock)
= OVS_LIST_INITIALIZER(&dpdk_mp_list);
-/* This mutex must be used by non pmd threads when allocating or freeing
+/* This spinlock must be used by non pmd threads when allocating or freeing
* mbufs through mempools. Since dpdk_queue_pkts() and dpdk_queue_flush() may
- * use mempools, a non pmd thread should hold this mutex while calling them */
-struct ovs_mutex nonpmd_mempool_mutex = OVS_MUTEX_INITIALIZER;
+ * use mempools, a non pmd thread should hold this spinlock while calling them
*/
+rte_spinlock_t nonpmd_mempool_spinlock = RTE_SPINLOCK_INITIALIZER;
struct dpdk_mp {
struct rte_mempool *mp;
int mtu;
int socket_id;
int refcount;
- struct ovs_list list_node OVS_GUARDED_BY(dpdk_mutex);
+ struct ovs_list list_node OVS_GUARDED_BY(dpdk_spinlock);
};
/* There should be one 'struct dpdk_tx_queue' created for
@@ -169,7 +178,7 @@ struct dpdk_tx_queue {
so we have to keep them around once they've been created
*/
-static struct ovs_list dpdk_ring_list OVS_GUARDED_BY(dpdk_mutex)
+static struct ovs_list dpdk_ring_list OVS_GUARDED_BY(dpdk_spinlock)
= OVS_LIST_INITIALIZER(&dpdk_ring_list);
struct dpdk_ring {
@@ -178,17 +187,18 @@ struct dpdk_ring {
struct rte_ring *cring_rx;
int user_port_id; /* User given port no, parsed from port name */
int eth_port_id; /* ethernet device port id */
- struct ovs_list list_node OVS_GUARDED_BY(dpdk_mutex);
+ struct ovs_list list_node OVS_GUARDED_BY(dpdk_spinlock);
};
struct netdev_dpdk {
struct netdev up;
int port_id;
int max_packet_len;
+ int type; /* DPDK or VHOST */
struct dpdk_tx_queue *tx_q;
- struct ovs_mutex mutex OVS_ACQ_AFTER(dpdk_mutex);
+ rte_spinlock_t spinlock OVS_ACQ_AFTER(dpdk_spinlock);
struct dpdk_mp *dpdk_mp;
int mtu;
@@ -202,8 +212,11 @@ struct netdev_dpdk {
struct rte_eth_link link;
int link_reset_cnt;
+ /* virtio-net structure for vhost device */
+ OVSRCU_TYPE(struct virtio_net *) virtio_dev;
+
/* In dpdk_list. */
- struct ovs_list list_node OVS_GUARDED_BY(dpdk_mutex);
+ struct ovs_list list_node OVS_GUARDED_BY(dpdk_spinlock);
rte_spinlock_t dpdkr_tx_lock;
};
@@ -216,14 +229,18 @@ static bool thread_is_pmd(void);
static int netdev_dpdk_construct(struct netdev *);
+void *start_cuse_session_loop(void *dummy);
+
+struct virtio_net * netdev_dpdk_get_virtio(const struct netdev_dpdk *dev);
+
static bool
is_dpdk_class(const struct netdev_class *class)
{
return class->construct == netdev_dpdk_construct;
}
-/* XXX: use dpdk malloc for entire OVS. infact huge page shld be used
- * for all other sengments data, bss and text. */
+/* XXX: use dpdk malloc for entire OVS. in fact huge page should be used
+ * for all other segments data, bss and text. */
static void *
dpdk_rte_mzalloc(size_t sz)
@@ -238,7 +255,7 @@ dpdk_rte_mzalloc(size_t sz)
}
/* XXX this function should be called only by pmd threads (or by non pmd
- * threads holding the nonpmd_mempool_mutex) */
+ * threads holding the nonpmd_mempool_spinlock) */
void
free_dpdk_buf(struct dp_packet *p)
{
@@ -289,7 +306,7 @@ ovs_rte_pktmbuf_init(struct rte_mempool *mp,
}
static struct dpdk_mp *
-dpdk_mp_get(int socket_id, int mtu) OVS_REQUIRES(dpdk_mutex)
+dpdk_mp_get(int socket_id, int mtu) OVS_REQUIRES(dpdk_spinlock)
{
struct dpdk_mp *dmp = NULL;
char mp_name[RTE_MEMPOOL_NAMESIZE];
@@ -377,13 +394,13 @@ dpdk_watchdog(void *dummy OVS_UNUSED)
pthread_detach(pthread_self());
for (;;) {
- ovs_mutex_lock(&dpdk_mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
LIST_FOR_EACH (dev, list_node, &dpdk_list) {
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
check_link_status(dev);
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
}
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dpdk_spinlock);
xsleep(DPDK_PORT_WATCHDOG_INTERVAL);
}
@@ -391,7 +408,7 @@ dpdk_watchdog(void *dummy OVS_UNUSED)
}
static int
-dpdk_eth_dev_init(struct netdev_dpdk *dev) OVS_REQUIRES(dpdk_mutex)
+dpdk_eth_dev_init(struct netdev_dpdk *dev) OVS_REQUIRES(dpdk_spinlock)
{
struct rte_pktmbuf_pool_private *mbp_priv;
struct ether_addr eth_addr;
@@ -484,15 +501,15 @@ netdev_dpdk_alloc_txq(struct netdev_dpdk *netdev,
unsigned int n_txqs)
static int
netdev_dpdk_init(struct netdev *netdev_, unsigned int port_no)
- OVS_REQUIRES(dpdk_mutex)
+ OVS_REQUIRES(dpdk_spinlock)
{
struct netdev_dpdk *netdev = netdev_dpdk_cast(netdev_);
int sid;
int err = 0;
- ovs_mutex_init(&netdev->mutex);
+ rte_spinlock_init(&netdev->spinlock);
- ovs_mutex_lock(&netdev->mutex);
+ rte_spinlock_lock(&netdev->spinlock);
/* If the 'sid' is negative, it means that the kernel fails
* to obtain the pci numa info. In that situation, always
@@ -501,6 +518,7 @@ netdev_dpdk_init(struct netdev *netdev_, unsigned int
port_no)
netdev->socket_id = sid < 0 ? SOCKET0 : sid;
netdev_dpdk_alloc_txq(netdev, NR_QUEUE);
netdev->port_id = port_no;
+ netdev->type = DPDK;
netdev->flags = 0;
netdev->mtu = ETHER_MTU;
netdev->max_packet_len = MTU_TO_MAX_LEN(netdev->mtu);
@@ -525,7 +543,7 @@ unlock:
if (err) {
rte_free(netdev->tx_q);
}
- ovs_mutex_unlock(&netdev->mutex);
+ rte_spinlock_unlock(&netdev->spinlock);
return err;
}
@@ -545,6 +563,65 @@ dpdk_dev_parse_name(const char dev_name[], const char
prefix[],
}
static int
+netdev_dpdk_vhost_construct(struct netdev *netdev_)
+{
+ struct netdev_dpdk *netdev = netdev_dpdk_cast(netdev_);
+ struct netdev_dpdk *tmp_netdev;
+ unsigned int port_no = 0;
+ int err = 0;
+ int sid = 0;
+
+ if (rte_eal_init_ret) {
+ return rte_eal_init_ret;
+ }
+ rte_spinlock_lock(&dpdk_spinlock);
+ rte_spinlock_init(&netdev->spinlock);
+ rte_spinlock_lock(&netdev->spinlock);
+
+ if (!list_is_empty(&dpdk_list)) {
+ LIST_FOR_EACH (tmp_netdev, list_node, &dpdk_list) {
+ if (tmp_netdev->type== VHOST) {
+ port_no++;
+ }
+ }
+ }
+
+ /* Retrieving socket id for master DPDK core */
+ sid = rte_lcore_to_socket_id(rte_get_master_lcore());
+ netdev->socket_id = sid < 0 ? SOCKET0 : sid;
+ netdev->port_id = port_no;
+ netdev->type = VHOST;
+
+ netdev->flags = 0;
+ netdev->mtu = ETHER_MTU;
+ netdev->max_packet_len = MTU_TO_MAX_LEN(netdev->mtu);
+
+ ovsrcu_set(&netdev->virtio_dev, NULL);
+ netdev->stats.rx_packets = 0;
+ netdev->stats.tx_packets = 0;
+
+ netdev->dpdk_mp = dpdk_mp_get(netdev->socket_id, netdev->mtu);
+ if (!netdev->dpdk_mp) {
+ err = ENOMEM;
+ goto unlock_dev;
+ }
+
+ netdev_->n_txq = NR_QUEUE;
+ netdev_->n_rxq = NR_QUEUE;
+
+ VLOG_INFO("%s is associated with VHOST port #%d\n", netdev_->name,
+ netdev->port_id);
+
+ list_push_back(&dpdk_list, &netdev->list_node);
+
+unlock_dev:
+ rte_spinlock_unlock(&netdev->spinlock);
+ rte_spinlock_unlock(&dpdk_spinlock);
+
+ return err;
+}
+
+static int
netdev_dpdk_construct(struct netdev *netdev)
{
unsigned int port_no;
@@ -560,9 +637,9 @@ netdev_dpdk_construct(struct netdev *netdev)
return err;
}
- ovs_mutex_lock(&dpdk_mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
err = netdev_dpdk_init(netdev, port_no);
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dpdk_spinlock);
return err;
}
@@ -571,17 +648,32 @@ netdev_dpdk_destruct(struct netdev *netdev_)
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
rte_eth_dev_stop(dev->port_id);
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
- ovs_mutex_lock(&dpdk_mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
rte_free(dev->tx_q);
list_remove(&dev->list_node);
dpdk_mp_put(dev->dpdk_mp);
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dpdk_spinlock);
+}
+
+static void
+netdev_dpdk_vhost_destruct(struct netdev *netdev_)
+{
+ struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
- ovs_mutex_destroy(&dev->mutex);
+ /* Can't remove a port while a guest is attached to it. */
+ if (netdev_dpdk_get_virtio(dev) != NULL) {
+ VLOG_ERR("Can not remove port, vhost device still attached\n");
+ return;
+ }
+
+ rte_spinlock_lock(&dpdk_spinlock);
+ list_remove(&dev->list_node);
+ dpdk_mp_put(dev->dpdk_mp);
+ rte_spinlock_unlock(&dpdk_spinlock);
}
static void
@@ -597,11 +689,11 @@ netdev_dpdk_get_config(const struct netdev *netdev_,
struct smap *args)
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
smap_add_format(args, "configured_rx_queues", "%d", netdev_->n_rxq);
smap_add_format(args, "configured_tx_queues", "%d", netdev_->n_txq);
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -628,19 +720,42 @@ netdev_dpdk_set_multiq(struct netdev *netdev_, unsigned
int n_txq,
return err;
}
- ovs_mutex_lock(&dpdk_mutex);
- ovs_mutex_lock(&netdev->mutex);
-
- rte_eth_dev_stop(netdev->port_id);
+ rte_spinlock_lock(&dpdk_spinlock);
+ rte_spinlock_lock(&netdev->spinlock);
netdev->up.n_txq = n_txq;
netdev->up.n_rxq = n_rxq;
+
+ rte_eth_dev_stop(netdev->port_id);
rte_free(netdev->tx_q);
netdev_dpdk_alloc_txq(netdev, n_txq);
err = dpdk_eth_dev_init(netdev);
- ovs_mutex_unlock(&netdev->mutex);
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&netdev->spinlock);
+ rte_spinlock_unlock(&dpdk_spinlock);
+
+ return err;
+}
+
+static int
+netdev_dpdk_vhost_set_multiq(struct netdev *netdev_, unsigned int n_txq,
+ unsigned int n_rxq)
+{
+ struct netdev_dpdk *netdev = netdev_dpdk_cast(netdev_);
+ int err = 0;
+
+ if (netdev->up.n_txq == n_txq && netdev->up.n_rxq == n_rxq) {
+ return err;
+ }
+
+ rte_spinlock_lock(&dpdk_spinlock);
+ rte_spinlock_lock(&netdev->spinlock);
+
+ netdev->up.n_txq = n_txq;
+ netdev->up.n_rxq = n_rxq;
+
+ rte_spinlock_unlock(&netdev->spinlock);
+ rte_spinlock_unlock(&dpdk_spinlock);
return err;
}
@@ -665,9 +780,9 @@ netdev_dpdk_rxq_construct(struct netdev_rxq *rxq_)
struct netdev_rxq_dpdk *rx = netdev_rxq_dpdk_cast(rxq_);
struct netdev_dpdk *netdev = netdev_dpdk_cast(rx->up.netdev);
- ovs_mutex_lock(&netdev->mutex);
+ rte_spinlock_lock(&netdev->spinlock);
rx->port_id = netdev->port_id;
- ovs_mutex_unlock(&netdev->mutex);
+ rte_spinlock_unlock(&netdev->spinlock);
return 0;
}
@@ -711,9 +826,9 @@ dpdk_queue_flush__(struct netdev_dpdk *dev, int qid)
for (i = nb_tx; i < txq->count; i++) {
rte_pktmbuf_free_seg(txq->burst_pkts[i]);
}
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
dev->stats.tx_dropped += txq->count-nb_tx;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
}
txq->count = 0;
@@ -731,6 +846,36 @@ dpdk_queue_flush(struct netdev_dpdk *dev, int qid)
dpdk_queue_flush__(dev, qid);
}
+/*
+ * The receive path for the vhost port is the TX path out from guest.
+ */
+static int
+netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq_,
+ struct dp_packet **packets, int *c)
+{
+ struct netdev_rxq_dpdk *rx = netdev_rxq_dpdk_cast(rxq_);
+ struct netdev *netdev = rx->up.netdev;
+ struct netdev_dpdk *vhost_dev = netdev_dpdk_cast(netdev);
+ struct virtio_net *virtio_dev = netdev_dpdk_get_virtio(vhost_dev);
+ int qid = 1;
+ uint16_t nb_rx = 0;
+
+ if (virtio_dev != NULL && (virtio_dev->flags & VIRTIO_DEV_RUNNING)) {
+ nb_rx = rte_vhost_dequeue_burst(virtio_dev, qid,
+ vhost_dev->dpdk_mp->mp, (struct rte_mbuf **)packets,
+ MAX_PKT_BURST);
+ if (!nb_rx) {
+ return EAGAIN;
+ }
+
+ vhost_dev->stats.rx_packets += (uint64_t)nb_rx;
+ }
+
+ *c = (int) nb_rx;
+
+ return 0;
+}
+
static int
netdev_dpdk_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet **packets,
int *c)
@@ -759,6 +904,44 @@ netdev_dpdk_rxq_recv(struct netdev_rxq *rxq_, struct
dp_packet **packets,
return 0;
}
+static int
+netdev_dpdk_vhost_send(struct netdev *netdev, int qid OVS_UNUSED, struct
dp_packet **pkts,
+ int cnt, bool may_steal)
+{
+ int i;
+ int tx_pkts;
+
+ struct netdev_dpdk *vhost_dev = netdev_dpdk_cast(netdev);
+ struct virtio_net *virtio_dev = netdev_dpdk_get_virtio(vhost_dev);
+
+ if (virtio_dev == NULL || !(virtio_dev->flags & VIRTIO_DEV_RUNNING)) {
+ VLOG_WARN_RL(&rl, "virtio_dev not added yet");
+
+ rte_spinlock_lock(&vhost_dev->spinlock);
+ vhost_dev->stats.tx_dropped+= cnt;
+ rte_spinlock_unlock(&vhost_dev->spinlock);
+ goto out;
+ }
+
+ /* Hardcoded until mq support */
+ qid = VIRTIO_RXQ;
+
+ tx_pkts = rte_vhost_enqueue_burst(virtio_dev, qid,
+ (struct rte_mbuf **)pkts, cnt);
+
+ vhost_dev->stats.tx_packets += tx_pkts;
+ vhost_dev->stats.tx_dropped += (cnt - tx_pkts);
+
+out:
+ if (may_steal) {
+ for (i = 0; i < cnt; i++) {
+ dp_packet_delete(pkts[i]);
+ }
+ }
+
+ return 0;
+}
+
inline static void
dpdk_queue_pkts(struct netdev_dpdk *dev, int qid,
struct rte_mbuf **pkts, int cnt)
@@ -800,11 +983,11 @@ dpdk_do_tx_copy(struct netdev *netdev, int qid, struct
dp_packet ** pkts,
int newcnt = 0;
int i;
- /* If we are on a non pmd thread we have to use the mempool mutex, because
+ /* If we are on a non pmd thread we have to use the mempool spinlock,
because
* every non pmd thread shares the same mempool cache */
if (!thread_is_pmd()) {
- ovs_mutex_lock(&nonpmd_mempool_mutex);
+ rte_spinlock_lock(&nonpmd_mempool_spinlock);
}
for (i = 0; i < cnt; i++) {
@@ -835,16 +1018,16 @@ dpdk_do_tx_copy(struct netdev *netdev, int qid, struct
dp_packet ** pkts,
}
if (OVS_UNLIKELY(dropped)) {
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
dev->stats.tx_dropped += dropped;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
}
dpdk_queue_pkts(dev, qid, mbufs, newcnt);
dpdk_queue_flush(dev, qid);
if (!thread_is_pmd()) {
- ovs_mutex_unlock(&nonpmd_mempool_mutex);
+ rte_spinlock_unlock(&nonpmd_mempool_spinlock);
}
}
@@ -893,9 +1076,9 @@ netdev_dpdk_send__(struct netdev_dpdk *dev, int qid,
}
if (OVS_UNLIKELY(dropped)) {
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
dev->stats.tx_dropped += dropped;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
}
}
}
@@ -916,12 +1099,12 @@ netdev_dpdk_set_etheraddr(struct netdev *netdev,
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
if (!eth_addr_equals(dev->hwaddr, mac)) {
memcpy(dev->hwaddr, mac, ETH_ADDR_LEN);
netdev_change_seq_changed(netdev);
}
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -932,9 +1115,9 @@ netdev_dpdk_get_etheraddr(const struct netdev *netdev,
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
memcpy(mac, dev->hwaddr, ETH_ADDR_LEN);
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -944,9 +1127,9 @@ netdev_dpdk_get_mtu(const struct netdev *netdev, int *mtup)
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
*mtup = dev->mtu;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -959,8 +1142,8 @@ netdev_dpdk_set_mtu(const struct netdev *netdev, int mtu)
struct dpdk_mp *old_mp;
struct dpdk_mp *mp;
- ovs_mutex_lock(&dpdk_mutex);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
+ rte_spinlock_lock(&dev->spinlock);
if (dev->mtu == mtu) {
err = 0;
goto out;
@@ -993,8 +1176,8 @@ netdev_dpdk_set_mtu(const struct netdev *netdev, int mtu)
dpdk_mp_put(old_mp);
netdev_change_seq_changed(netdev);
out:
- ovs_mutex_unlock(&dev->mutex);
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dev->spinlock);
+ rte_spinlock_unlock(&dpdk_spinlock);
return err;
}
@@ -1002,6 +1185,44 @@ static int
netdev_dpdk_get_carrier(const struct netdev *netdev_, bool *carrier);
static int
+netdev_dpdk_vhost_get_stats(const struct netdev *netdev,
+ struct netdev_stats *stats)
+{
+ struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
+
+ rte_spinlock_lock(&dev->spinlock);
+ memset(stats, 0, sizeof(*stats));
+ /* Unsupported Stats */
+ stats->rx_errors = UINT64_MAX;
+ stats->tx_errors = UINT64_MAX;
+ stats->multicast = UINT64_MAX;
+ stats->collisions = UINT64_MAX;
+ stats->rx_crc_errors = UINT64_MAX;
+ stats->rx_fifo_errors = UINT64_MAX;
+ stats->rx_frame_errors = UINT64_MAX;
+ stats->rx_length_errors = UINT64_MAX;
+ stats->rx_missed_errors= UINT64_MAX;
+ stats->rx_over_errors= UINT64_MAX;
+ stats->tx_aborted_errors= UINT64_MAX;
+ stats->tx_carrier_errors= UINT64_MAX;
+ stats->tx_errors= UINT64_MAX;
+ stats->tx_fifo_errors= UINT64_MAX;
+ stats->tx_heartbeat_errors= UINT64_MAX;
+ stats->tx_window_errors= UINT64_MAX;
+ stats->rx_bytes += UINT64_MAX;
+ stats->rx_dropped += UINT64_MAX;
+ stats->tx_bytes += UINT64_MAX;
+
+ /* Supported Stats */
+ stats->rx_packets += dev->stats.rx_packets;
+ stats->tx_packets += dev->stats.tx_packets;
+ stats->tx_dropped += dev->stats.tx_dropped;
+ rte_spinlock_unlock(&dev->spinlock);
+
+ return 0;
+}
+
+static int
netdev_dpdk_get_stats(const struct netdev *netdev, struct netdev_stats *stats)
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
@@ -1009,7 +1230,7 @@ netdev_dpdk_get_stats(const struct netdev *netdev, struct
netdev_stats *stats)
bool gg;
netdev_dpdk_get_carrier(netdev, &gg);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
rte_eth_stats_get(dev->port_id, &rte_stats);
memset(stats, 0, sizeof(*stats));
@@ -1023,7 +1244,7 @@ netdev_dpdk_get_stats(const struct netdev *netdev, struct
netdev_stats *stats)
stats->multicast = rte_stats.imcasts;
stats->tx_dropped = dev->stats.tx_dropped;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -1038,9 +1259,9 @@ netdev_dpdk_get_features(const struct netdev *netdev_,
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
struct rte_eth_link link;
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
link = dev->link;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
if (link.link_duplex == ETH_LINK_AUTONEG_DUPLEX) {
if (link.link_speed == ETH_LINK_SPEED_AUTONEG) {
@@ -1080,9 +1301,9 @@ netdev_dpdk_get_ifindex(const struct netdev *netdev)
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev);
int ifindex;
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
ifindex = dev->port_id;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return ifindex;
}
@@ -1092,10 +1313,31 @@ netdev_dpdk_get_carrier(const struct netdev *netdev_,
bool *carrier)
{
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
+
check_link_status(dev);
*carrier = dev->link.link_status;
- ovs_mutex_unlock(&dev->mutex);
+
+ rte_spinlock_unlock(&dev->spinlock);
+
+ return 0;
+}
+
+static int
+netdev_dpdk_vhost_get_carrier(const struct netdev *netdev_, bool *carrier)
+{
+ struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
+ struct virtio_net *virtio_dev = netdev_dpdk_get_virtio(dev);
+
+ rte_spinlock_lock(&dev->spinlock);
+
+ if (virtio_dev != NULL && (virtio_dev->flags & VIRTIO_DEV_RUNNING)) {
+ *carrier =1;
+ } else {
+ *carrier = 0 ;
+ }
+
+ rte_spinlock_unlock(&dev->spinlock);
return 0;
}
@@ -1106,9 +1348,9 @@ netdev_dpdk_get_carrier_resets(const struct netdev
*netdev_)
struct netdev_dpdk *dev = netdev_dpdk_cast(netdev_);
long long int carrier_resets;
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
carrier_resets = dev->link_reset_cnt;
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
return carrier_resets;
}
@@ -1123,7 +1365,7 @@ netdev_dpdk_set_miimon(struct netdev *netdev_ OVS_UNUSED,
static int
netdev_dpdk_update_flags__(struct netdev_dpdk *dev,
enum netdev_flags off, enum netdev_flags on,
- enum netdev_flags *old_flagsp)
OVS_REQUIRES(dev->mutex)
+ enum netdev_flags *old_flagsp)
OVS_REQUIRES(dev->spinlock)
{
int err;
@@ -1139,18 +1381,20 @@ netdev_dpdk_update_flags__(struct netdev_dpdk *dev,
return 0;
}
- if (dev->flags & NETDEV_UP) {
- err = rte_eth_dev_start(dev->port_id);
- if (err)
- return -err;
- }
+ if (dev->type == DPDK) {
+ if (dev->flags & NETDEV_UP) {
+ err = rte_eth_dev_start(dev->port_id);
+ if (err)
+ return -err;
+ }
- if (dev->flags & NETDEV_PROMISC) {
- rte_eth_promiscuous_enable(dev->port_id);
- }
+ if (dev->flags & NETDEV_PROMISC) {
+ rte_eth_promiscuous_enable(dev->port_id);
+ }
- if (!(dev->flags & NETDEV_UP)) {
- rte_eth_dev_stop(dev->port_id);
+ if (!(dev->flags & NETDEV_UP)) {
+ rte_eth_dev_stop(dev->port_id);
+ }
}
return 0;
@@ -1164,9 +1408,9 @@ netdev_dpdk_update_flags(struct netdev *netdev_,
struct netdev_dpdk *netdev = netdev_dpdk_cast(netdev_);
int error;
- ovs_mutex_lock(&netdev->mutex);
+ rte_spinlock_lock(&netdev->spinlock);
error = netdev_dpdk_update_flags__(netdev, off, on, old_flagsp);
- ovs_mutex_unlock(&netdev->mutex);
+ rte_spinlock_unlock(&netdev->spinlock);
return error;
}
@@ -1180,9 +1424,9 @@ netdev_dpdk_get_status(const struct netdev *netdev_,
struct smap *args)
if (dev->port_id < 0)
return ENODEV;
- ovs_mutex_lock(&dev->mutex);
+ rte_spinlock_lock(&dev->spinlock);
rte_eth_dev_info_get(dev->port_id, &dev_info);
- ovs_mutex_unlock(&dev->mutex);
+ rte_spinlock_unlock(&dev->spinlock);
smap_add_format(args, "driver_name", "%s", dev_info.driver_name);
@@ -1206,7 +1450,7 @@ netdev_dpdk_get_status(const struct netdev *netdev_,
struct smap *args)
static void
netdev_dpdk_set_admin_state__(struct netdev_dpdk *dev, bool admin_state)
- OVS_REQUIRES(dev->mutex)
+ OVS_REQUIRES(dev->spinlock)
{
enum netdev_flags old_flags;
@@ -1237,9 +1481,9 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn,
int argc,
if (netdev && is_dpdk_class(netdev->netdev_class)) {
struct netdev_dpdk *dpdk_dev = netdev_dpdk_cast(netdev);
- ovs_mutex_lock(&dpdk_dev->mutex);
+ rte_spinlock_lock(&dpdk_dev->spinlock);
netdev_dpdk_set_admin_state__(dpdk_dev, up);
- ovs_mutex_unlock(&dpdk_dev->mutex);
+ rte_spinlock_unlock(&dpdk_dev->spinlock);
netdev_close(netdev);
} else {
@@ -1250,17 +1494,149 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn,
int argc,
} else {
struct netdev_dpdk *netdev;
- ovs_mutex_lock(&dpdk_mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
LIST_FOR_EACH (netdev, list_node, &dpdk_list) {
- ovs_mutex_lock(&netdev->mutex);
+ rte_spinlock_lock(&netdev->spinlock);
netdev_dpdk_set_admin_state__(netdev, up);
- ovs_mutex_unlock(&netdev->mutex);
+ rte_spinlock_unlock(&netdev->spinlock);
}
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dpdk_spinlock);
}
unixctl_command_reply(conn, "OK");
}
+/*
+ * Set virtqueue flags so that we do not receive interrupts.
+ */
+static void
+set_irq_status(struct virtio_net *dev)
+{
+ dev->virtqueue[VIRTIO_RXQ]->used->flags = VRING_USED_F_NO_NOTIFY;
+ dev->virtqueue[VIRTIO_TXQ]->used->flags = VRING_USED_F_NO_NOTIFY;
+}
+
+/*
+ * A new virtio-net device is added to a vhost port.
+ */
+static int
+new_device(struct virtio_net *dev)
+{
+ struct netdev_dpdk *netdev;
+ bool exists = false;
+
+ /* Disable notifications. */
+ set_irq_status(dev);
+ dev->flags |= VIRTIO_DEV_RUNNING;
+
+ rte_spinlock_lock(&dpdk_spinlock);
+ /* Add device to the vhost port with the same name as that passed down. */
+ LIST_FOR_EACH(netdev, list_node, &dpdk_list) {
+ if (strncmp(dev->ifname, netdev->up.name, IFNAMSIZ) == 0) {
+ rte_spinlock_lock(&netdev->spinlock);
+ ovsrcu_set(&netdev->virtio_dev, dev);
+ rte_spinlock_unlock(&netdev->spinlock);
+ exists = true;
+ break;
+ }
+ }
+ rte_spinlock_unlock(&dpdk_spinlock);
+
+ if (!exists) {
+ VLOG_ERR("(%ld) VHOST Device can't be added - name not found \n",
+ dev->device_fh);
+ return -1;
+ }
+
+ VLOG_INFO("(%ld) VHOST Device has been added \n", dev->device_fh);
+
+ return 0;
+}
+
+/*
+ * Remove a virtio-net device from the specific vhost port. Use dev->remove
+ * flag to stop any more packets from being sent or received to/from a VM and
+ * ensure all currently queued packets have been sent/received before removing
+ * the device.
+ */
+static void
+destroy_device(volatile struct virtio_net *dev)
+{
+ struct netdev_dpdk *vhost_dev;
+ dev->flags &= ~VIRTIO_DEV_RUNNING;
+
+ rte_spinlock_lock(&dpdk_spinlock);
+ LIST_FOR_EACH (vhost_dev, list_node, &dpdk_list) {
+ if (netdev_dpdk_get_virtio(vhost_dev) == dev) {
+
+ /*
+ * Wait for other threads to quiesce before
+ * setting the virtio_dev to NULL.
+ */
+ ovsrcu_synchronize();
+
+ rte_spinlock_lock(&vhost_dev->spinlock);
+ ovsrcu_set(&vhost_dev->virtio_dev, NULL);
+ rte_spinlock_unlock(&vhost_dev->spinlock);
+
+ ovsrcu_quiesce_start();
+ }
+ }
+ rte_spinlock_unlock(&dpdk_spinlock);
+
+ VLOG_INFO("%ld Vhost Device has been removed\n", dev->device_fh);
+}
+
+struct virtio_net *
+netdev_dpdk_get_virtio(const struct netdev_dpdk *dev)
+{
+ return ovsrcu_get(struct virtio_net *, &dev->virtio_dev);
+}
+
+
+/*
+ * These callbacks allow virtio-net devices to be added to vhost ports when
+ * configuration has been fully complete.
+ */
+const struct virtio_net_device_ops virtio_net_device_ops =
+{
+ .new_device = new_device,
+ .destroy_device = destroy_device,
+};
+
+static int
+dpdk_vhost_class_init(void)
+{
+ int err = -1;
+
+ rte_vhost_driver_callback_register(&virtio_net_device_ops);
+
+ /* Register CUSE device to handle IOCTLs.
+ * Unless otherwise specified on the vswitchd command line, dev_basename
+ * is set to vhost-net.
+ */
+ err = rte_vhost_driver_register(dev_basename);
+
+ if (err != 0) {
+ VLOG_ERR("CUSE device setup failure.\n");
+ return -1;
+ }
+
+ ovs_thread_create("cuse_thread", start_cuse_session_loop, NULL);
+
+ return 0;
+}
+
+void *
+start_cuse_session_loop(void *dummy OVS_UNUSED)
+{
+ pthread_detach(pthread_self());
+
+ ovsrcu_quiesce_start();
+ rte_vhost_driver_session_start();
+
+ return NULL;
+}
+
static void
dpdk_common_init(void)
{
@@ -1330,7 +1706,7 @@ dpdk_ring_create(const char dev_name[], unsigned int
port_no,
}
static int
-dpdk_ring_open(const char dev_name[], unsigned int *eth_port_id)
OVS_REQUIRES(dpdk_mutex)
+dpdk_ring_open(const char dev_name[], unsigned int *eth_port_id)
OVS_REQUIRES(dpdk_spinlock)
{
struct dpdk_ring *ivshmem;
unsigned int port_no;
@@ -1377,7 +1753,7 @@ netdev_dpdk_ring_construct(struct netdev *netdev)
return rte_eal_init_ret;
}
- ovs_mutex_lock(&dpdk_mutex);
+ rte_spinlock_lock(&dpdk_spinlock);
err = dpdk_ring_open(netdev->name, &port_no);
if (err) {
@@ -1387,11 +1763,12 @@ netdev_dpdk_ring_construct(struct netdev *netdev)
err = netdev_dpdk_init(netdev, port_no);
unlock_dpdk:
- ovs_mutex_unlock(&dpdk_mutex);
+ rte_spinlock_unlock(&dpdk_spinlock);
return err;
}
-#define NETDEV_DPDK_CLASS(NAME, INIT, CONSTRUCT, MULTIQ, SEND) \
+#define NETDEV_DPDK_CLASS(NAME, INIT, CONSTRUCT, DESTRUCT, MULTIQ, SEND, \
+ GET_CARRIER, GET_STATS, GET_FEATURES, GET_STATUS, RXQ_RECV) \
{ \
NAME, \
INIT, /* init */ \
@@ -1400,14 +1777,14 @@ unlock_dpdk:
\
netdev_dpdk_alloc, \
CONSTRUCT, \
- netdev_dpdk_destruct, \
+ DESTRUCT, \
netdev_dpdk_dealloc, \
netdev_dpdk_get_config, \
NULL, /* netdev_dpdk_set_config */ \
NULL, /* get_tunnel_config */ \
- NULL, /* build header */ \
- NULL, /* push header */ \
- NULL, /* pop header */ \
+ NULL, /* build header */ \
+ NULL, /* push header */ \
+ NULL, /* pop header */ \
netdev_dpdk_get_numa_id, /* get_numa_id */ \
MULTIQ, /* set_multiq */ \
\
@@ -1419,11 +1796,11 @@ unlock_dpdk:
netdev_dpdk_get_mtu, \
netdev_dpdk_set_mtu, \
netdev_dpdk_get_ifindex, \
- netdev_dpdk_get_carrier, \
+ GET_CARRIER, \
netdev_dpdk_get_carrier_resets, \
netdev_dpdk_set_miimon, \
- netdev_dpdk_get_stats, \
- netdev_dpdk_get_features, \
+ GET_STATS, \
+ GET_FEATURES, \
NULL, /* set_advertisements */ \
\
NULL, /* set_policing */ \
@@ -1445,7 +1822,7 @@ unlock_dpdk:
NULL, /* get_in6 */ \
NULL, /* add_router */ \
NULL, /* get_next_hop */ \
- netdev_dpdk_get_status, \
+ GET_STATUS, \
NULL, /* arp_lookup */ \
\
netdev_dpdk_update_flags, \
@@ -1454,7 +1831,7 @@ unlock_dpdk:
netdev_dpdk_rxq_construct, \
netdev_dpdk_rxq_destruct, \
netdev_dpdk_rxq_dealloc, \
- netdev_dpdk_rxq_recv, \
+ RXQ_RECV, \
NULL, /* rx_wait */ \
NULL, /* rxq_drain */ \
}
@@ -1463,16 +1840,43 @@ int
dpdk_init(int argc, char **argv)
{
int result;
+ int base = 0;
if (argc < 2 || strcmp(argv[1], "--dpdk"))
return 0;
- /* Make sure program name passed to rte_eal_init() is vswitchd. */
+ /* Ignore the --dpdk argument, but keep the program name argument as this
+ * is needed later for a call to rte_eal_init()
+ */
argv[1] = argv[0];
argc--;
argv++;
+ /* If the basename parameter has been provided, set 'dev_basename' to
+ * this string if it meets the correct criteria. Otherwise, set it to the
+ * default (vhost-net).
+ */
+
+ if (strcmp(argv[1], "--basename")==0 &&
+ (sizeof(argv[2]) <= MAX_BASENAME_SZ)) {
+ dev_basename = argv[2];
+
+ /* Remove the basename configuration parameters from the argument
+ * list, so that the correct elements are passed to the DPDK
+ * initialization function (including the program name as the first).
+ */
+ argv[2] = argv[0];
+ argc -= 2;
+ argv += 2; /* Increment by two to bypass the basename arguments */
+ base = 2;
+
+ VLOG_INFO("User-provided basename in use: /dev/%s\n", dev_basename);
+ } else {
+ dev_basename = "vhost-net";
+ VLOG_INFO("No basename provided - defaulting to /dev/vhost-net\n");
+ }
+
/* Make sure things are initialized ... */
result = rte_eal_init(argc, argv);
if (result < 0) {
@@ -1489,7 +1893,7 @@ dpdk_init(int argc, char **argv)
/* We are called from the main thread here */
thread_set_nonpmd();
- return result + 1;
+ return result + 1 + base;
}
const struct netdev_class dpdk_class =
@@ -1497,16 +1901,42 @@ const struct netdev_class dpdk_class =
"dpdk",
NULL,
netdev_dpdk_construct,
+ netdev_dpdk_destruct,
netdev_dpdk_set_multiq,
- netdev_dpdk_eth_send);
+ netdev_dpdk_eth_send,
+ netdev_dpdk_get_carrier,
+ netdev_dpdk_get_stats,
+ netdev_dpdk_get_features,
+ netdev_dpdk_get_status,
+ netdev_dpdk_rxq_recv);
const struct netdev_class dpdk_ring_class =
NETDEV_DPDK_CLASS(
"dpdkr",
NULL,
netdev_dpdk_ring_construct,
+ netdev_dpdk_destruct,
+ NULL,
+ netdev_dpdk_ring_send,
+ netdev_dpdk_get_carrier,
+ netdev_dpdk_get_stats,
+ netdev_dpdk_get_features,
+ netdev_dpdk_get_status,
+ netdev_dpdk_rxq_recv);
+
+const struct netdev_class dpdk_vhost_class =
+ NETDEV_DPDK_CLASS(
+ "dpdkvhost",
+ dpdk_vhost_class_init,
+ netdev_dpdk_vhost_construct,
+ netdev_dpdk_vhost_destruct,
+ netdev_dpdk_vhost_set_multiq,
+ netdev_dpdk_vhost_send,
+ netdev_dpdk_vhost_get_carrier,
+ netdev_dpdk_vhost_get_stats,
+ NULL,
NULL,
- netdev_dpdk_ring_send);
+ netdev_dpdk_vhost_rxq_recv);
void
netdev_dpdk_register(void)
@@ -1521,6 +1951,7 @@ netdev_dpdk_register(void)
dpdk_common_init();
netdev_register_provider(&dpdk_class);
netdev_register_provider(&dpdk_ring_class);
+ netdev_register_provider(&dpdk_vhost_class);
ovsthread_once_done(&once);
}
}
diff --git a/lib/netdev.c b/lib/netdev.c
index b76da13..149b39a 100644
--- a/lib/netdev.c
+++ b/lib/netdev.c
@@ -108,7 +108,8 @@ bool
netdev_is_pmd(const struct netdev *netdev)
{
return (!strcmp(netdev->netdev_class->type, "dpdk") ||
- !strcmp(netdev->netdev_class->type, "dpdkr"));
+ !strcmp(netdev->netdev_class->type, "dpdkr") ||
+ !strcmp(netdev->netdev_class->type, "dpdkvhost"));
}
static void
diff --git a/utilities/automake.mk b/utilities/automake.mk
index 6fd7176..0ff0a63 100644
--- a/utilities/automake.mk
+++ b/utilities/automake.mk
@@ -43,7 +43,8 @@ EXTRA_DIST += \
utilities/ovs-save \
utilities/ovs-tcpundump.in \
utilities/ovs-test.in \
- utilities/ovs-vlan-test.in
+ utilities/ovs-vlan-test.in \
+ utilities/qemu-wrap.py
MAN_ROOTS += \
utilities/ovs-appctl.8.in \
utilities/ovs-benchmark.1.in \
diff --git a/utilities/qemu-wrap.py b/utilities/qemu-wrap.py
new file mode 100755
index 0000000..5cee849
--- /dev/null
+++ b/utilities/qemu-wrap.py
@@ -0,0 +1,389 @@
+#!/usr/bin/python
+#
+# BSD LICENSE
+#
+# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+#####################################################################
+# This script is designed to modify the call to the QEMU emulator
+# to support userspace vhost when starting a guest machine through
+# libvirt with vhost enabled. The steps to enable this are as follows
+# and should be run as root:
+#
+# 1. Place this script in a libvirtd's binary search PATH ($PATH)
+# A good location would be in the same directory that the QEMU
+# binary is located
+#
+# 2. Ensure that the script has the same owner/group and file
+# permissions as the QEMU binary
+#
+# 3. Update the VM xml file using "virsh edit VM.xml"
+#
+# 3.a) Set the VM to use the launch script
+#
+# Set the emulator path contained in the
+# <emulator><emulator/> tags
+#
+# e.g replace <emulator>/usr/bin/qemu-kvm<emulator/>
+# with <emulator>/usr/bin/qemu-wrap.py<emulator/>
+#
+# 3.b) Set the VM's device's to use vhost-net offload
+#
+# <interface type="network">
+# <model type="virtio"/>
+# <driver name="vhost"/>
+# <interface/>
+#
+# 4. Enable libvirt to access our userpace device file by adding it to
+# controllers cgroup for libvirtd using the following steps
+#
+# 4.a) In /etc/libvirt/qemu.conf add/edit the following lines:
+# 1) cgroup_controllers = [ ... "devices", ... ]
+# 2) clear_emulator_capabilities = 0
+# 3) user = "root"
+# 4) group = "root"
+# 5) cgroup_device_acl = [
+# "/dev/null", "/dev/full", "/dev/zero",
+# "/dev/random", "/dev/urandom",
+# "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
+# "/dev/rtc", "/dev/hpet", "/dev/net/tun",
+# "/dev/<devbase-name>-<index>",
+# "/dev/hugepages"
+# ]
+#
+# 4.b) Disable SELinux or set to permissive mode
+#
+# 4.c) Mount cgroup device controller
+# "mkdir /dev/cgroup"
+# "mount -t cgroup none /dev/cgroup -o devices"
+#
+# 4.d) Set hugetlbfs_mount variable - ( Optional )
+# VMs using userspace vhost must use hugepage backed
+# memory. This can be enabled in the libvirt XML
+# config by adding a memory backing section to the
+# XML config e.g.
+# <memoryBacking>
+# <hugepages/>
+# </memoryBacking>
+# This memory backing section should be added after the
+# <memory> and <currentMemory> sections. This will add
+# flags "-mem-prealloc -mem-path <path>" to the QEMU
+# command line. The hugetlbfs_mount variable can be used
+# to override the default <path> passed through by libvirt.
+#
+# if "-mem-prealloc" or "-mem-path <path>" are not passed
+# through and a vhost device is detected then these options will
+# be automatically added by this script. This script will detect
+# the system hugetlbfs mount point to be used for <path>. The
+# default <path> for this script can be overidden by the
+# hugetlbfs_dir variable in the configuration section of this script.
+#
+#
+# 4.e) Restart the libvirtd system process
+# e.g. on Fedora "systemctl restart libvirtd.service"
+#
+#
+# 4.f) Edit the Configuration Parameters section of this script
+# to point to the correct emulator location and set any
+# addition options
+#
+# The script modifies the libvirtd Qemu call by modifying/adding
+# options based on the configuration parameters below.
+# NOTE:
+# emul_path and us_vhost_path must be set
+# All other parameters are optional
+#####################################################################
+
+
+#############################################
+# Configuration Parameters
+#############################################
+#Path to QEMU binary
+emul_path = "/usr/local/bin/qemu-system-x86_64"
+
+#Path to userspace vhost device file
+# This filename should match the --dev-basename --dev-index parameters of
+# the command used to launch the userspace vhost sample application e.g.
+# if the sample app lauch command is:
+# ./build/vhost-switch ..... --dev-basename usvhost --dev-index 1
+# then this variable should be set to:
+# us_vhost_path = "/dev/usvhost-1"
+us_vhost_path = "/dev/usvhost-1"
+
+#List of additional user defined emulation options. These options will
+#be added to all Qemu calls
+emul_opts_user = []
+
+#List of additional user defined emulation options for vhost only.
+#These options will only be added to vhost enabled guests
+emul_opts_user_vhost = []
+
+#For all VHOST enabled VMs, the VM memory is preallocated from hugetlbfs
+# Set this variable to one to enable this option for all VMs
+use_huge_all = 0
+
+#Instead of autodetecting, override the hugetlbfs directory by setting
+#this variable
+hugetlbfs_dir = ""
+
+#############################################
+
+
+#############################################
+# ****** Do Not Modify Below this Line ******
+#############################################
+
+import sys, os, subprocess
+import time
+import signal
+
+
+#List of open userspace vhost file descriptors
+fd_list = []
+
+#additional virtio device flags when using userspace vhost
+vhost_flags = [ "csum=off",
+ "gso=off",
+ "guest_tso4=off",
+ "guest_tso6=off",
+ "guest_ecn=off"
+ ]
+
+#String of the path to the Qemu process pid
+qemu_pid = "/tmp/%d-qemu.pid" % os.getpid()
+
+#############################################
+# Signal haldler to kill Qemu subprocess
+#############################################
+def kill_qemu_process(signum, stack):
+ pidfile = open(qemu_pid, 'r')
+ pid = int(pidfile.read())
+ os.killpg(pid, signal.SIGTERM)
+ pidfile.close()
+
+
+#############################################
+# Find the system hugefile mount point.
+# Note:
+# if multiple hugetlbfs mount points exist
+# then the first one found will be used
+#############################################
+def find_huge_mount():
+
+ if (len(hugetlbfs_dir)):
+ return hugetlbfs_dir
+
+ huge_mount = ""
+
+ if (os.access("/proc/mounts", os.F_OK)):
+ f = open("/proc/mounts", "r")
+ line = f.readline()
+ while line:
+ line_split = line.split(" ")
+ if line_split[2] == 'hugetlbfs':
+ huge_mount = line_split[1]
+ break
+ line = f.readline()
+ else:
+ print "/proc/mounts not found"
+ exit (1)
+
+ f.close
+ if len(huge_mount) == 0:
+ print "Failed to find hugetlbfs mount point"
+ exit (1)
+
+ return huge_mount
+
+
+#############################################
+# Get a userspace Vhost file descriptor
+#############################################
+def get_vhost_fd():
+
+ if (os.access(us_vhost_path, os.F_OK)):
+ fd = os.open( us_vhost_path, os.O_RDWR)
+ else:
+ print ("US-Vhost file %s not found" %us_vhost_path)
+ exit (1)
+
+ return fd
+
+
+#############################################
+# Check for vhostfd. if found then replace
+# with our own vhost fd and append any vhost
+# flags onto the end
+#############################################
+def modify_netdev_arg(arg):
+
+ global fd_list
+ vhost_in_use = 0
+ s = ''
+ new_opts = []
+ netdev_opts = arg.split(",")
+
+ for opt in netdev_opts:
+ #check if vhost is used
+ if "vhost" == opt[:5]:
+ vhost_in_use = 1
+ else:
+ new_opts.append(opt)
+
+ #if using vhost append vhost options
+ if vhost_in_use == 1:
+ #append vhost on option
+ new_opts.append('vhost=on')
+ #append vhostfd ption
+ new_fd = get_vhost_fd()
+ new_opts.append('vhostfd=' + str(new_fd))
+ fd_list.append(new_fd)
+
+ #concatenate all options
+ for opt in new_opts:
+ if len(s) > 0:
+ s+=','
+
+ s+=opt
+
+ return s
+
+
+#############################################
+# Main
+#############################################
+def main():
+
+ global fd_list
+ global vhost_in_use
+ new_args = []
+ num_cmd_args = len(sys.argv)
+ emul_call = ''
+ mem_prealloc_set = 0
+ mem_path_set = 0
+ num = 0;
+
+ #parse the parameters
+ while (num < num_cmd_args):
+ arg = sys.argv[num]
+
+ #Check netdev +1 parameter for vhostfd
+ if arg == '-netdev':
+ num_vhost_devs = len(fd_list)
+ new_args.append(arg)
+
+ num+=1
+ arg = sys.argv[num]
+ mod_arg = modify_netdev_arg(arg)
+ new_args.append(mod_arg)
+
+ #append vhost flags if this is a vhost device
+ # and -device is the next arg
+ # i.e -device -opt1,-opt2,...,-opt3,%vhost
+ if (num_vhost_devs < len(fd_list)):
+ num+=1
+ arg = sys.argv[num]
+ if arg == '-device':
+ new_args.append(arg)
+ num+=1
+ new_arg = sys.argv[num]
+ for flag in vhost_flags:
+ new_arg = ''.join([new_arg,',',flag])
+ new_args.append(new_arg)
+ else:
+ new_args.append(arg)
+ elif arg == '-mem-prealloc':
+ mem_prealloc_set = 1
+ new_args.append(arg)
+ elif arg == '-mem-path':
+ mem_path_set = 1
+ new_args.append(arg)
+
+ else:
+ new_args.append(arg)
+
+ num+=1
+
+ #Set Qemu binary location
+ emul_call+=emul_path
+ emul_call+=" "
+
+ #Add prealloc mem options if using vhost and not already added
+ if ((len(fd_list) > 0) and (mem_prealloc_set == 0)):
+ emul_call += "-mem-prealloc "
+
+ #Add mempath mem options if using vhost and not already added
+ if ((len(fd_list) > 0) and (mem_path_set == 0)):
+ #Detect and add hugetlbfs mount point
+ mp = find_huge_mount()
+ mp = "".join(["-mem-path ", mp])
+ emul_call += mp
+ emul_call += " "
+
+ #add user options
+ for opt in emul_opts_user:
+ emul_call += opt
+ emul_call += " "
+
+ #Add add user vhost only options
+ if len(fd_list) > 0:
+ for opt in emul_opts_user_vhost:
+ emul_call += opt
+ emul_call += " "
+
+ #Add updated libvirt options
+ iter_args = iter(new_args)
+ #skip 1st arg i.e. call to this script
+ next(iter_args)
+ for arg in iter_args:
+ emul_call+=str(arg)
+ emul_call+= " "
+
+ emul_call += "-pidfile %s " % qemu_pid
+ #Call QEMU
+ process = subprocess.Popen(emul_call, shell=True, preexec_fn=os.setsid)
+
+ for sig in [signal.SIGTERM, signal.SIGINT, signal.SIGHUP, signal.SIGQUIT]:
+ signal.signal(sig, kill_qemu_process)
+
+ process.wait()
+
+ #Close usvhost files
+ for fd in fd_list:
+ os.close(fd)
+ #Cleanup temporary files
+ if os.access(qemu_pid, os.F_OK):
+ os.remove(qemu_pid)
+
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vswitchd/ovs-vswitchd.c b/vswitchd/ovs-vswitchd.c
index 812c00b..822d53e 100644
--- a/vswitchd/ovs-vswitchd.c
+++ b/vswitchd/ovs-vswitchd.c
@@ -252,7 +252,9 @@ usage(void)
daemon_usage();
vlog_usage();
printf("\nDPDK options:\n"
- " --dpdk options Initialize DPDK datapath.\n");
+ " --dpdk options Initialize DPDK datapath.\n"
+ " --basename BASENAME override default character device name\n"
+ " for use with userspace vHost.\n");
printf("\nOther options:\n"
" --unixctl=SOCKET override default control socket name\n"
" -h, --help display this help message\n"
--
1.7.4.1
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev
