Just a quick update that a fix has been sent to dpdk.org for review. This
turns out to be a problem with vmxnet3 dpdk pmd driver when queue number are
reconfigured. Thanks Daniele for the reproduction, code review and test.
Sothy, please let us know if applying the patch to dpdk get your problem
resolved (you are already copied on the review request to dpdk.org).
Yong
On 7/28/16, 12:43 AM, "sothy shan" <sothy....@gmail.com> wrote:
On Thu, Jul 28, 2016 at 2:29 AM, Yong Wang <yongw...@vmware.com> wrote:
>
>> On Jul 27, 2016, at 12:25 AM, sothy shan <sothy....@gmail.com> wrote:
>>
>> On Tue, Jul 26, 2016 at 6:32 PM, Mauricio Vasquez
>> <mauricio.vasq...@polito.it> wrote:
>>> Hello Sothy,
>>>
>>> Sorry for the delay in answering, some busy days here.
>>>
>>>
>>> On 07/25/2016 12:03 PM, sothy shan wrote:
>>>>
>>>> Hello Mauricio,
>>>>
>>>> On Sun, Jul 24, 2016 at 12:17 PM, Mauricio Vasquez
>>>> <mauricio.vasq...@polito.it> wrote:
>>>>>
>>>>> Hi Sothy,
>>>>>
>>>>>
>>>>> On 07/21/2016 11:55 AM, sothy shan wrote:
>>>>>>
>>>>>> On Thu, Jul 21, 2016 at 11:43 AM, Mauricio Vasquez
>>>>>> <mauricio.vasq...@polito.it> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/21/2016 10:51 AM, sothy shan wrote:
>>>>>>>>
>>>>>>>> On Thu, Jul 21, 2016 at 10:41 AM, sothy shan <sothy....@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Wed, Jul 20, 2016 at 9:02 PM, Mauricio Vasquez
>>>>>>>>> <mauricio.vasq...@polito.it> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Sothy,
>>>>>>>>>>
>>>>>>>>>> On 07/20/2016 07:40 PM, Aaron Conole wrote:
>>>>>>>>>>>
>>>>>>>>>>> sothy shan <sothy....@gmail.com> writes:
>>>>>>>>>>>
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>
>>>>>>>>>>> Hi Sothy,
>>>>>>>>>>>
>>>>>>>>>>>> I followed the installion guide provided in the web site
>>>>>>>>>>>>
>>>>>>>>>>>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openvswitch_ovs_blob_master_INSTALL.DPDK.md-23ovstc&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=ojiXK7Tj1ANsHE_AlchVROP4R7LrgLQqVqTx7BjJQas&e=
>>>>>>>>>>>> to install dpdk OVS.
>>>>>>>>>>>>
>>>>>>>>>>>> I am rurnning Ubuntu 14.04 (kernel 3.13). It is a VM running
>>>>>>>>>>>> VMware.
>>>>>>>>>>>> One intereface is enabled by VMXNET3.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It is not clear for me, are you trying to install OvS within the
VM?
>>>>>>>>>>
>>>>>>>>>>>> I used dpdk 16.04
>>>>>>>>>>>> Today I made clone of OVS (committ
>>>>>>>>>>>> :bf32e3e2c029da18b5d6fd9210cb0ea12a1d0383).
>>>>>>>>>>>>
>>>>>>>>>>>> The following are different configuration compared to above
link.
>>>>>>>>>>>>
>>>>>>>>>>>> sysctl -w vm.nr_hugepages=2048
>>>>>>>>>>>> Binded interface via igb_uio.
>>>>>>>>>>>>
>>>>>>>>>>>> Then followed same configuration to start ovs-vsswitch with
dpdk.
>>>>>>>>>>>
>>>>>>>>>>> Can you please post the steps exactly that you followed? Which
>>>>>>>>>>> user
>>>>>>>>>>> you
>>>>>>>>>>> were running as, etc? Can you also post how you expect packets
to
>>>>>>>>>>> arrive into the vm?
>>>>>>>>>>>
>>>>>>>>>>>> I am able to creatre bridge. When I create a port. It gives
error
>>>>>>>>>>>> message.
>>>>>>>>>>>
>>>>>>>>>>> Please show the steps for this, too. The exact commands you
used.
>>>>>>>>>>>
>>>>>>>>>>>> When I check ovs-vswitchd.log file,
>>>>>>>>>>>>
>>>>>>>>>>>> There is warning that dpif_netlink|WARN|genric netlink family,
>>>>>>>>>>>> ovs_datapath does not exist. The openvswitch kernel module may
be
>>>>>>>>>>>> not
>>>>>>>>>>>> loaded.
>>>>>>>>>>>
>>>>>>>>>>> This is definitely a problem. How did you start the
ovs-vswitchd
>>>>>>>>>>> daemon?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is not a problem Aaron, the ovs kernel module is not needed
>>>>>>>>>> when
>>>>>>>>>> ovs is
>>>>>>>>>> compiled with DPDK.
>>>>>>>>>>>>
>>>>>>>>>>>> Then ERR: ERR|interface dpdk0 start error:operation not
permitted.
>>>>>>>>>>>>
>>>>>>>>>>>> The corresponding port is loaded by dpdk driver as shown in
>>>>>>>>>>>> dpdk_nic_bind --status.
>>>>>>>>>>>>
>>>>>>>>>>>> After that I did sudo ovs-vsctl show. it show br and port
details
>>>>>>>>>>>> as
>>>>>>>>>>>> well.
>>>>>>>>>>>> I am able to add one flow rule as well. But ovs bridge didnt
get
>>>>>>>>>>>> packets from the interface.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you know why this error is popup? anyway to solve the
problem?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you very much
>>>>>>>>>>>>
>>>>>>>>>>>> Best regars
>>>>>>>>>>>> Sothy
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> discuss mailing list
>>>>>>>>>>> discuss@openvswitch.org
>>>>>>>>>>>
https://urldefense.proofpoint.com/v2/url?u=http-3A__openvswitch.org_mailman_listinfo_discuss&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=DZpyuDSP-5BYH4bvfNHk68XRFB9-M-M43FxZCCpZKho&e=
>>>>>>>>>>
>>>>>>>>>> As Aaron wrote, having the exact commands is important to help
you.
>>>>>>>>>
>>>>>>>>> My configuration and log info:
>>>>>>>>>
>>>>>>>>> sudo sysctl vm.nr_hugepages=2048
>>>>>>>>> sudo mkdir -p /dev/hugepages
>>>>>>>>> sudo mount -t hugetlbfs hugetlbfs /dev/hugepages
>>>>>>>>> sudo modprobe uio
>>>>>>>>> sudo insmod $DPDK_BUILD/kmod/igb_uio.ko
>>>>>>>>> sudo $DPDK_DIR/tools/dpdk_nic_bind.py --status
>>>>>>>>> sudo $DPDK_DIR/tools/dpdk_nic_bind.py -b igb_uio 0000:0b:00.0
>>>>>>>>> sudo ./dpdk-16.04/tools/dpdk_nic_bind.py --status
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Network devices using DPDK-compatible driver
>>>>>>>>> ============================================
>>>>>>>>> 0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=
>>>>>>>>> 0000:13:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=
>>>>>>>>>
>>>>>>>>> Network devices using kernel driver
>>>>>>>>> ===================================
>>>>>>>>> 0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3
>>>>>>>>> unused=igb_uio *Active*
>>>>>>>>>
>>>>>>>>> Other network devices
>>>>>>>>> =====================
>>>>>>>>> <none>
>>>>>>>>>
>>>>>>>>> .........Then......................
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> sudo ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>>>>>>>>> sudo ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>>>>>>
>>>>>>> These two commands are inverted.
>>>>>>> The bridge has to be created first and then ports have to be added
to
>>>>>>> it.
>>>>>>>
>>>>>> It is my mistake when coping.
>>>>>>>>>
>>>>>>>>> Log file/
>>>>>>>>> 2016-07-21T07:53:35.527Z|00001|vlog|INFO|opened log file
>>>>>>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>>>>>>> 2016-07-21T07:53:35.529Z|00002|ovs_numa|INFO|Discovered 2 CPU
cores
>>>>>>>>> on
>>>>>>>>> NUMA node 0
>>>>>>>>> 2016-07-21T07:53:35.529Z|00003|ovs_numa|INFO|Discovered 1 NUMA
nodes
>>>>>>>>> and 2 CPU cores
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
2016-07-21T07:53:35.529Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>>>>> connecting...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
2016-07-21T07:53:35.529Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>>>>> connected
>>>>>>>>> 2016-07-21T07:53:35.533Z|00006|dpdk|INFO|DPDK Enabled,
initializing
>>>>>>>>> 2016-07-21T07:53:35.533Z|00007|dpdk|INFO|No vhost-sock-dir
provided -
>>>>>>>>> defaulting to /usr/local/var/run/openvswitch
>>>>>>>>> 2016-07-21T07:53:35.533Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>>>>>>> --socket-mem 1024,0 -c 0x00000001
>>>>>>>>> 2016-07-21T07:53:36.606Z|00009|timeval|WARN|Unreasonably long
1076ms
>>>>>>>>> poll interval (80ms user, 875ms system)
>>>>>>>>> 2016-07-21T07:53:36.606Z|00010|timeval|WARN|faults: 3306 minor, 0
>>>>>>>>> major
>>>>>>>>> 2016-07-21T07:53:36.606Z|00011|timeval|WARN|context switches: 3
>>>>>>>>> voluntary, 101 involuntary
>>>>>>>>> 2016-07-21T07:53:36.606Z|00012|coverage|INFO|Event coverage, avg
rate
>>>>>>>>> over last: 5 seconds, last minute, last hour, hash=398361de:
>>>>>>>>> 2016-07-21T07:53:36.606Z|00013|coverage|INFO|bridge_reconfigure
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.606Z|00014|coverage|INFO|cmap_expand
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 9
>>>>>>>>> 2016-07-21T07:53:36.606Z|00015|coverage|INFO|miniflow_malloc
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 9
>>>>>>>>> 2016-07-21T07:53:36.606Z|00016|coverage|INFO|hmap_pathological
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.606Z|00017|coverage|INFO|hmap_expand
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 640
>>>>>>>>> 2016-07-21T07:53:36.606Z|00018|coverage|INFO|txn_unchanged
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 2
>>>>>>>>> 2016-07-21T07:53:36.606Z|00019|coverage|INFO|txn_incomplete
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00020|coverage|INFO|poll_create_node
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 35
>>>>>>>>> 2016-07-21T07:53:36.607Z|00021|coverage|INFO|seq_change
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 38
>>>>>>>>> 2016-07-21T07:53:36.607Z|00022|coverage|INFO|pstream_open
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00023|coverage|INFO|stream_open
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00024|coverage|INFO|util_xalloc
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 11049
>>>>>>>>> 2016-07-21T07:53:36.607Z|00025|coverage|INFO|netdev_get_hwaddr
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00026|coverage|INFO|netlink_received
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 3
>>>>>>>>> 2016-07-21T07:53:36.607Z|00027|coverage|INFO|netlink_sent
>>>>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>>>>> 2016-07-21T07:53:36.607Z|00028|coverage|INFO|86 events never hit
>>>>>>>>> 2016-07-21T07:53:36.607Z|00029|bridge|INFO|ovs-vswitchd (Open
>>>>>>>>> vSwitch)
>>>>>>>>> 2.5.90
>>>>>>>>> 2016-07-21T07:54:50.861Z|00030|memory|INFO|18980 kB peak resident
set
>>>>>>>>> size after 75.3 seconds
>>>>>>>>>
2016-07-21T07:54:50.866Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath supports recirculation
>>>>>>>>>
2016-07-21T07:54:50.866Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> MPLS label stack length probed as 3
>>>>>>>>>
2016-07-21T07:54:50.866Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath supports truncate action
>>>>>>>>>
2016-07-21T07:54:50.866Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath supports unique flow ids
>>>>>>>>>
2016-07-21T07:54:50.866Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath does not support ct_state
>>>>>>>>>
2016-07-21T07:54:50.866Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath does not support ct_zone
>>>>>>>>>
2016-07-21T07:54:50.866Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath does not support ct_mark
>>>>>>>>>
2016-07-21T07:54:50.866Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath does not support ct_label
>>>>>>>>>
2016-07-21T07:54:50.866Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>>>>> Datapath does not support ct_state_nat
>>>>>>>>> 2016-07-21T07:54:50.875Z|00040|bridge|INFO|bridge br0: added
>>>>>>>>> interface
>>>>>>>>> br0 on port 65534
>>>>>>>>> 2016-07-21T07:54:50.887Z|00041|dpif_netlink|WARN|Generic Netlink
>>>>>>>>> family 'ovs_datapath' does not exist. The Open vSwitch kernel
module
>>>>>>>>> is probably not loaded.
>>>>>>>>> 2016-07-21T07:54:50.887Z|00042|bridge|INFO|bridge br0: using
datapath
>>>>>>>>> ID 0000c664fbd86348
>>>>>>>>> 2016-07-21T07:54:50.887Z|00043|connmgr|INFO|br0: added service
>>>>>>>>> controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>>>>>>> 2016-07-21T07:58:18.796Z|00044|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>>>>> 2016-07-21T07:58:18.810Z|00045|dpdk|ERR|Interface dpdk0 start
error:
>>>>>>>>> Operation not permitted
>>>>>>>>> 2016-07-21T07:58:18.810Z|00046|dpif|WARN|netdev@ovs-netdev:
failed to
>>>>>>>>> add dpdk0 as port: Operation not permitted
>>>>>>>>> 2016-07-21T08:10:35.440Z|00047|connmgr|INFO|br0<->unix: 1
flow_mods
>>>>>>>>> in
>>>>>>>>> the last 0 s (1 adds)
>>>>>>>
>>>>>>> Could you also provide the full terminal output when you run
>>>>>>> ovs-vswitchd?,
>>>>>>> I would like to see the DPDK output
>>>>>>
>>>>>> vrouter@vrouter-virtual-machine:~/ovs$ sudo ovs-vswitchd
unix:$DB_SOCK
>>>>>> --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>>>>> 2016-07-21T09:33:35Z|00001|vlog|INFO|opened log file
>>>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>>>> 2016-07-21T09:33:35Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on
NUMA
>>>>>> node 0
>>>>>> 2016-07-21T09:33:35Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and
2
>>>>>> CPU
>>>>>> cores
>>>>>>
>>>>>>
>>>>>>
2016-07-21T09:33:35Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>> connecting...
>>>>>>
>>>>>>
>>>>>>
2016-07-21T09:33:35Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>>>> connected
>>>>>> 2016-07-21T09:33:35Z|00006|dpdk|INFO|DPDK Enabled, initializing
>>>>>> 2016-07-21T09:33:35Z|00007|dpdk|INFO|No vhost-sock-dir provided -
>>>>>> defaulting to /usr/local/var/run/openvswitch
>>>>>> 2016-07-21T09:33:35Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>>>> --socket-mem 1024,0 -c 0x00000001
>>>>>> EAL: Detected lcore 0 as core 0 on socket 0
>>>>>> EAL: Detected lcore 1 as core 0 on socket 0
>>>>>> EAL: Support maximum 128 logical core(s) by configuration.
>>>>>> EAL: Detected 2 lcore(s)
>>>>>> EAL: Probing VFIO support...
>>>>>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>>>>>> directory)
>>>>>> EAL: VFIO modules not loaded, skipping VFIO support...
>>>>>> EAL: Setting up physically contiguous memory...
>>>>>> EAL: Ask a virtual area of 0x5e400000 bytes
>>>>>> EAL: Virtual area found at 0x7fa618400000 (size = 0x5e400000)
>>>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>>>> EAL: Virtual area found at 0x7fa618000000 (size = 0x200000)
>>>>>> EAL: Ask a virtual area of 0xa1800000 bytes
>>>>>> EAL: Virtual area found at 0x7fa576600000 (size = 0xa1800000)
>>>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>>>> EAL: Virtual area found at 0x7fa576200000 (size = 0x200000)
>>>>>> EAL: Requesting 512 pages of size 2MB from socket 0
>>>>>> EAL: TSC frequency is ~2799213 KHz
>>>>>> EAL: Master lcore 0 is ready (tid=78ad7b80;cpuset=[0])
>>>>>> EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL: Not managed by a supported kernel driver, skipped
>>>>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>>>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL: PCI memory mapped at 0x7fa658400000
>>>>>> EAL: PCI memory mapped at 0x7fa658401000
>>>>>> EAL: PCI memory mapped at 0x7fa658402000
>>>>>> EAL: PCI device 0000:13:00.0 on NUMA socket -1
>>>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>>>> EAL: PCI memory mapped at 0x7fa658404000
>>>>>> EAL: PCI memory mapped at 0x7fa658405000
>>>>>> EAL: PCI memory mapped at 0x7fa658406000
>>>>>> Zone 0: name:<RG_MP_log_history>, phys:0x82bfdec0, len:0x2080,
>>>>>> virt:0x7fa6583fdec0, socket_id:0, flags:0
>>>>>> Zone 1: name:<MP_log_history>, phys:0x82973d40, len:0x28a0c0,
>>>>>> virt:0x7fa658173d40, socket_id:0, flags:0
>>>>>> Zone 2: name:<rte_eth_dev_data>, phys:0x82943200, len:0x2f800,
>>>>>> virt:0x7fa658143200, socket_id:0, flags:0
>>>>>> 2016-07-21T09:33:36Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath supports recirculation
>>>>>> 2016-07-21T09:33:36Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
>>>>>> label stack length probed as 3
>>>>>> 2016-07-21T09:33:36Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath supports truncate action
>>>>>> 2016-07-21T09:33:36Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath supports unique flow ids
>>>>>> 2016-07-21T09:33:36Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath does not support ct_state
>>>>>> 2016-07-21T09:33:36Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath does not support ct_zone
>>>>>> 2016-07-21T09:33:36Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath does not support ct_mark
>>>>>> 2016-07-21T09:33:36Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath does not support ct_label
>>>>>> 2016-07-21T09:33:36Z|00017|ofproto_dpif|INFO|netdev@ovs-netdev:
>>>>>> Datapath does not support ct_state_nat
>>>>>> 2016-07-21T09:33:36Z|00018|bridge|INFO|bridge br0: added interface
br0
>>>>>> on port 65534
>>>>>> 2016-07-21T09:33:36Z|00019|dpif_netlink|WARN|Generic Netlink family
>>>>>> 'ovs_datapath' does not exist. The Open vSwitch kernel module is
>>>>>> probably not loaded.
>>>>>> 2016-07-21T09:33:36Z|00020|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>> 2016-07-21T09:33:36Z|00021|dpdk|ERR|Interface dpdk0 start error:
>>>>>> Operation not permitted
>>>>>> 2016-07-21T09:33:36Z|00022|bridge|INFO|bridge br0: using datapath ID
>>>>>> 0000c664fbd86348
>>>>>> 2016-07-21T09:33:36Z|00023|connmgr|INFO|br0: added service controller
>>>>>> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>>>> 2016-07-21T09:33:36Z|00024|timeval|WARN|Unreasonably long 1244ms poll
>>>>>> interval (233ms user, 880ms system)
>>>>>> 2016-07-21T09:33:36Z|00025|timeval|WARN|faults: 3902 minor, 0 major
>>>>>> 2016-07-21T09:33:36Z|00026|timeval|WARN|context switches: 10
>>>>>> voluntary, 214 involuntary
>>>>>> 2016-07-21T09:33:36Z|00027|coverage|INFO|Event coverage, avg rate
over
>>>>>> last: 5 seconds, last minute, last hour, hash=75cfd2fc:
>>>>>> 2016-07-21T09:33:36Z|00028|coverage|INFO|bridge_reconfigure
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00029|coverage|INFO|ofproto_flush
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00030|coverage|INFO|ofproto_update_port
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00031|coverage|INFO|rev_flow_table
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00032|coverage|INFO|cmap_expand
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 22
>>>>>> 2016-07-21T09:33:36Z|00033|coverage|INFO|cmap_shrink
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 7
>>>>>> 2016-07-21T09:33:36Z|00034|coverage|INFO|dpif_port_add
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 2
>>>>>> 2016-07-21T09:33:36Z|00035|coverage|INFO|dpif_flow_flush
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 2
>>>>>> 2016-07-21T09:33:36Z|00036|coverage|INFO|dpif_flow_get
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 5
>>>>>> 2016-07-21T09:33:36Z|00037|coverage|INFO|dpif_flow_put
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 10
>>>>>> 2016-07-21T09:33:36Z|00038|coverage|INFO|dpif_flow_del
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 5
>>>>>> 2016-07-21T09:33:36Z|00039|coverage|INFO|dpif_execute
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 3
>>>>>> 2016-07-21T09:33:36Z|00040|coverage|INFO|flow_extract
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 4
>>>>>> 2016-07-21T09:33:36Z|00041|coverage|INFO|miniflow_malloc
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 16
>>>>>> 2016-07-21T09:33:36Z|00042|coverage|INFO|hmap_pathological
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00043|coverage|INFO|hmap_expand
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 653
>>>>>> 2016-07-21T09:33:36Z|00044|coverage|INFO|netdev_get_stats
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00045|coverage|INFO|poll_create_node
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 15
>>>>>> 2016-07-21T09:33:36Z|00046|coverage|INFO|poll_zero_timeout
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00047|coverage|INFO|seq_change
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 575
>>>>>> 2016-07-21T09:33:36Z|00048|coverage|INFO|pstream_open
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 3
>>>>>> 2016-07-21T09:33:36Z|00049|coverage|INFO|stream_open
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00050|coverage|INFO|util_xalloc
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 11820
>>>>>> 2016-07-21T09:33:36Z|00051|coverage|INFO|netdev_set_policing
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00052|coverage|INFO|netdev_get_ifindex
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00053|coverage|INFO|netdev_get_hwaddr
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 2
>>>>>> 2016-07-21T09:33:36Z|00054|coverage|INFO|netdev_set_hwaddr
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00055|coverage|INFO|netdev_get_ethtool
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 2
>>>>>> 2016-07-21T09:33:36Z|00056|coverage|INFO|netlink_received
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 6
>>>>>> 2016-07-21T09:33:36Z|00057|coverage|INFO|netlink_recv_jumbo
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 1
>>>>>> 2016-07-21T09:33:36Z|00058|coverage|INFO|netlink_sent
>>>>>> 0.0/sec 0.000/sec 0.0000/sec total: 5
>>>>>> 2016-07-21T09:33:36Z|00059|coverage|INFO|70 events never hit
>>>>>> 2016-07-21T09:33:36Z|00060|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>>>> 2016-07-21T09:33:36Z|00061|dpdk|ERR|Interface dpdk0 start error:
>>>>>> Operation not permitted
>>>>>
>>>>> The PMD start function is failing, unfortunately that error does not
tell
>>>>> that much information.
>>>>>
>>>>> I would suggest you to enable the debug of the vmxnet3 PMD in DPDK,
set
>>>>> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y in the config/common_base file
of
>>>>> DPDK, then recompile DPDK and OvS.
>>>>>
>>>>> Please try once again and send the terminal output, I hope we can get
a
>>>>> more
>>>>> information of what is going on with that log.
>>>>
>>>> Based on your request, I reproduce the log with
>>>> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
>>>> sudo ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock
>>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile --detach
>>>> 2016-07-25T09:24:17Z|00001|vlog|INFO|opened log file
>>>> /var/log/openvswitch/ovs-vswitchd.log
>>>> 2016-07-25T09:24:17Z|00002|ovs_numa|INFO|Discovered 2 CPU cores on NUMA
>>>> node 0
>>>> 2016-07-25T09:24:17Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 2
CPU
>>>> cores
>>>>
>>>>
2016-07-25T09:24:17Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>> connecting...
>>>>
>>>>
2016-07-25T09:24:17Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
>>>> connected
>>>> 2016-07-25T09:24:17Z|00006|dpdk|INFO|DPDK Enabled, initializing
>>>> 2016-07-25T09:24:17Z|00007|dpdk|INFO|No vhost-sock-dir provided -
>>>> defaulting to /usr/local/var/run/openvswitch
>>>> 2016-07-25T09:24:17Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd
>>>> --socket-mem 1024,0 -c 0x00000001
>>>> EAL: Detected lcore 0 as core 0 on socket 0
>>>> EAL: Detected lcore 1 as core 0 on socket 0
>>>> EAL: Support maximum 128 logical core(s) by configuration.
>>>> EAL: Detected 2 lcore(s)
>>>> EAL: Probing VFIO support...
>>>> EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
>>>> directory)
>>>> EAL: VFIO modules not loaded, skipping VFIO support...
>>>> EAL: Setting up physically contiguous memory...
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fdea6000000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0x5e000000 bytes
>>>> EAL: Virtual area found at 0x7fde47e00000 (size = 0x5e000000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fde47a00000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fde47600000 (size = 0x200000)
>>>> EAL: Ask a virtual area of 0xa1800000 bytes
>>>> EAL: Virtual area found at 0x7fdda5c00000 (size = 0xa1800000)
>>>> EAL: Ask a virtual area of 0x200000 bytes
>>>> EAL: Virtual area found at 0x7fdda5800000 (size = 0x200000)
>>>> EAL: Requesting 512 pages of size 2MB from socket 0
>>>> EAL: TSC frequency is ~2799209 KHz
>>>> EAL: Master lcore 0 is ready (tid=a855eb80;cpuset=[0])
>>>> PMD: rte_vmxnet3_pmd_init(): >>
>>>> EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL: Not managed by a supported kernel driver, skipped
>>>> EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL: PCI memory mapped at 0x7fdea6200000
>>>> EAL: PCI memory mapped at 0x7fdea6201000
>>>> EAL: PCI memory mapped at 0x7fdea6202000
>>>> PMD: eth_vmxnet3_dev_init(): >>
>>>> PMD: eth_vmxnet3_dev_init(): Hardware version : 1
>>>> PMD: eth_vmxnet3_dev_init(): UPT hardware version : 1
>>>> PMD: eth_vmxnet3_dev_init(): MAC Address : 00:0c:29:b5:9f:5b
>>>> EAL: PCI device 0000:13:00.0 on NUMA socket -1
>>>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>>>> EAL: Not managed by a supported kernel driver, skipped
>>>> Zone 0: name:<RG_MP_log_history>, phys:0x555fdec0, len:0x2080,
>>>> virt:0x7fdea61fdec0, socket_id:0, flags:0
>>>> Zone 1: name:<MP_log_history>, phys:0x95375f00, len:0x28a0c0,
>>>> virt:0x7fde87975f00, socket_id:0, flags:0
>>>> Zone 2: name:<rte_eth_dev_data>, phys:0x555cd300, len:0x2f800,
>>>> virt:0x7fdea61cd300, socket_id:0, flags:0
>>>>
>>>> PMD: vmxnet3_dev_configure(): >>
>>>> PMD: vmxnet3_dev_tx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_rx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_start(): >>
>>>> PMD: vmxnet3_rss_configure(): >>
>>>> PMD: vmxnet3_write_mac(): Writing MAC Address : 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_disable_intr(): >>
>>>> PMD: vmxnet3_dev_rxtx_init(): >>
>>>> 2016-07-25T09:40:23Z|00020|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_dev_stop(): >>
>>>> PMD: vmxnet3_disable_intr(): >>
>>>> PMD: vmxnet3_dev_stop(): Device reset.
>>>> PMD: vmxnet3_dev_clear_queues(): >>
>>>> PMD: vmxnet3_dev_configure(): >>
>>>> PMD: vmxnet3_dev_tx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_tx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_tx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_rx_queue_setup(): >>
>>>> PMD: vmxnet3_dev_start(): >>
>>>> PMD: vmxnet3_rss_configure(): >>
>>>> PMD: vmxnet3_write_mac(): Writing MAC Address : 00:0c:29:b5:9f:5b
>>>> PMD: vmxnet3_dev_start(): Device activation: UNSUCCESSFUL
>>>
>>>
>>> Unfortunately it is the most I can help you, it is strange for me why
>>> test-pmd is working and why OvS is not.
>>> I think it is a DPDK issue, then the next step is to post on the DPDK
>>> mailing (d...@dpdk.org, add also in CC Yong Wang <yongw...@vmware.com>
that
>>> is the vmxnet3 maintainer), send a link to this thread in order to be
clear
>>> what you have tested.
>>
>>
>> Hello Yong Wang,
>>
>> I am running fedora VM in ESXi 5.5.0 environment. I tried to install
>> OVS and DPDK. OVS couldnt start DPDK device.
>> You can see the error in the mail below. Do you know why? IF you need
>> more information. I can provide more information.
>>
>> Good news, when I tried in Fedora 23 VM in ESXi 5.5.0, it works. It is
>> possible to bind an dpdk device and get packets on port.
>>
>> Thanks Mauricio for your help and reply.
>>
>> Best regards
>> Sothy
>
> Sothy,
>
Hello,
> You mentioned that on Fedora 23, it works. Then on some version Fedora
that it failed?
Initially, I started working on Ubuntu 14.04. It produce error.
Therefore, I changed in Fedora. I worked directly on Fedora 23.
For fedora, I only tried on Fedora 23. It works. I didnt tried on
other version. All Vms are in ESX 5.5. The problem is in Ubuntu vm
ONLY,
> From your previous post, you mentioned that it’s a Ubuntu 14.04 VM? Can
you clarify? Other than the changes you >mentioned already, are there any
other changes (such as ring buffer size, number of rx queues, etc) that have
been >changed?
I didnt do any changes. I just followed basic commands given in OVDK
install.md and
>Is there any updates installed on your ESX 5.5?
no
>If yes, what’s the exact update version? Can you also send me a copy of
your vmkernel.log offline?
I will send you offline.
>
> Also copying Daniele as he is trying to help reproduce the reported issue.
>
>>
>>>
>>>> I deleted exisiting bridge and recreated::::::
>>>>
>>>>
>>>> Then I added bridge. After I try to create bridge. It gives error. The
>>>> erros is same as shown in log file.
>>>>
>>>> 2016-07-25T09:29:43.248Z|00020|bridge|INFO|bridge br0: added interface
>>>> br0 on port 65534
>>>> 2016-07-25T09:29:43.253Z|00021|dpif_netlink|WARN|Generic Netlink
>>>> family 'ovs_datapath' does not exist. The Open vSwitch kernel module
>>>> is probably not loaded.
>>>> 2016-07-25T09:29:43.253Z|00022|bridge|INFO|bridge br0: using datapath
>>>> ID 0000e6aac740584b
>>>> 2016-07-25T09:29:43.253Z|00023|connmgr|INFO|br0: added service
>>>> controller "punix:/usr/local/var/run/openvswitch/br0.mgmt"
>>>> 2016-07-25T09:30:54.008Z|00024|dpdk|INFO|Port 0: 00:0c:29:b5:9f:5b
>>>> 2016-07-25T09:30:54.021Z|00025|dpdk|ERR|Interface dpdk0 start error:
>>>> Operation not permitted
>>>> 2016-07-25T09:30:54.021Z|00026|dpif|WARN|netdev@ovs-netdev: failed to
>>>> add dpdk0 as port: Operation not permitted
>>>>
>>>>
>>>>
>>>>> Another thing to do would be to bind the NICs to the uio_pci_generic
and
>>>>> VFIO modules, just to be sure that the issue is also present there.
>>>>
>>>> I tried in uio and uio_pci_genric also. Both are giving same error.
>>>> I suspect I cannt activate VFIO in VM since it is BIOS suport also. AM
I
>>>> right?
>>>>
>>>> Thanks
>>>>
>>>>>
>>>>>> in command line some information for your verification:
>>>>>>
>>>>>> sudo ovs-vsctl show
>>>>>> d53612e0-5d82-4cc1-af7d-ee887a737ef2
>>>>>> Bridge "br0"
>>>>>> Port "dpdk0"
>>>>>> Interface "dpdk0"
>>>>>> type: dpdk
>>>>>> Port "br0"
>>>>>> Interface "br0"
>>>>>> type: internal
>>>>>> vrouter@vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl del-br br0
>>>>>> vrouter@vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl add-br br0 --
>>>>>> set bridge br0 datapath_type=netdev
>>>>>> vrouter@vrouter-virtual-machine:~/ovs$ sudo ovs-vsctl add-port br0
>>>>>> dpdk0 -- set Interface dpdk0 type=dpdk
>>>>>> ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd
>>>>>> log for details.
>>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> sudo ovs-vsctl show
>>>>>>>>> d53612e0-5d82-4cc1-af7d-ee887a737ef2
>>>>>>>>> Bridge "br0"
>>>>>>>>> Port "dpdk0"
>>>>>>>>> Interface "dpdk0"
>>>>>>>>> type: dpdk
>>>>>>>>> Port "br0"
>>>>>>>>> Interface "br0"
>>>>>>>>> type: internal
>>>>>>>>>
>>>>>>>>> sudo ovs-ofctl dump-flows br0
>>>>>>>>> NXST_FLOW reply (xid=0x4):
>>>>>>>>> cookie=0x0, duration=487.638s, table=0, n_packets=0, n_bytes=0,
>>>>>>>>> idle_age=487, priority=0 actions=NORMAL
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> In addition I would recommend to use the test-pmd [1] application
>>>>>>>>>> from
>>>>>>>>>> DPDK
>>>>>>>>>> to check if the issue is also present in that case.
>>>>>>>>
>>>>>>>> I run testpmd command.The result is
>>>>>>>>
>>>>>>>> Configuring Port 0 (socket 0)
>>>>>>>> Port 0: 00:0C:29:B5:9F:5B
>>>>>>>> Configuring Port 1 (socket 0)
>>>>>>>> Port 1: 00:0C:29:B5:9F:65
>>>>>>>> Checking link statuses...
>>>>>>>> Port 0 Link Up - speed 10000 Mbps - full-duplex
>>>>>>>> Port 1 Link Up - speed 10000 Mbps - full-duplex
>>>>>>>> Done
>>>>>>>> No commandline core given, start packet forwarding
>>>>>>>> io packet forwarding - CRC stripping disabled -
packets/burst=32
>>>>>>>> nb forwarding cores=1 - nb forwarding ports=2
>>>>>>>> RX queues=1 - RX desc=128 - RX free threshold=0
>>>>>>>> RX threshold registers: pthresh=0 hthresh=0 wthresh=0
>>>>>>>> TX queues=1 - TX desc=512 - TX free threshold=0
>>>>>>>> TX threshold registers: pthresh=0 hthresh=0 wthresh=0
>>>>>>>> TX RS bit threshold=0 - TXQ flags=0x200
>>>>>>>> Press enter to exit
>>>>>>>>
>>>>>>>> Telling cores to stop...
>>>>>>>> Waiting for lcores to finish...
>>>>>>>>
>>>>>>>> ---------------------- Forward statistics for port 0
>>>>>>>> ----------------------
>>>>>>>> RX-packets: 88 RX-dropped: 0 RX-total:
88
>>>>>>>> TX-packets: 2 TX-dropped: 0 TX-total:
2
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
----------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> ---------------------- Forward statistics for port 1
>>>>>>>> ----------------------
>>>>>>>> RX-packets: 2 RX-dropped: 0 RX-total:
2
>>>>>>>> TX-packets: 88 TX-dropped: 0 TX-total:
88
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
----------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> +++++++++++++++ Accumulated forward statistics for all
>>>>>>>> ports+++++++++++++++
>>>>>>>> RX-packets: 90 RX-dropped: 0 RX-total:
90
>>>>>>>> TX-packets: 90 TX-dropped: 0 TX-total:
90
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>
>>>>>>>> make sense?
>>>>>>>
>>>>>>>
>>>>>>> Yes, it makes sense. Ports are being recognized by DPDK and some
>>>>>>> packets
>>>>>>> were exchanged.
>>>>>>>
>>>>>>>
>>>>>>>>>> [1]
https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk.org_doc_guides-2D16.04_testpmd-5Fapp-5Fug_index.html&d=CwIBaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=44mSO5N5yEs4CeCdtQE0xt0F7J0p67_mApYVAzyYms0&m=vpaHcL64W3vOTY9P8yRZOdCJzbL_oWzZH6J2u4iT614&s=JnLeHoyKuiecYI95-fcfFiuev5g2CDweT2MnyUiKWzM&e=
>>>>>>>>>>
>>>>>>>>>> Mauricio Vasquez,
>>>>>>>
>>>>>>>
>>>>> Mauricio V,
>>>
>>>
>
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
