Hi, I am following the
guide: http://openvswitch.org/support/dist-docs-2.5/INSTALL.DPDK.md.html
to deploy ovs + dpdk + qemu
At last I find that, I can not ping br0 on host in the VM
if I dump the packages with ovs-ofctl
* port 3: rx pkts=127, bytes=5646, drop=?, errs=0, frame=?, over=?,
crc=?*
* tx pkts=0, bytes=0, drop=194, errs=?, coll=?*
I found that vhost-user1 port, it can receive packages from VM, so rx
is not zero.
But it can not send packages to VM, so tx is zero, all packages are droped
So did you guys face this problem? how to fix that?
-----------------------detailed steps--------------------------
1. set the environment
[root@localhost ~]# cat dpdkrc
export DPDK_DIR=/root/dpdk-2.2.0
export RTE_SDK=/root/dpdk-2.2.0
export RTE_TARGET=x86_64-native-linuxapp-gcc
export DESTDIR=/usr/local/bin/dpdk
export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc
export OVS_DIR=/root/openvswitch-2.5.0
2. build the dpdk
cd to /root/dpdk-2.2.0
|sed
||'s/CONFIG_RTE_BUILD_COMBINE_LIBS=n/CONFIG_RTE_BUILD_COMBINE_LIBS=y/'||-||i
config||/||common_linuxapp|
|
|
||make install T||=||x86_64||-||native||-||linuxapp||-||gcc||
||
||
3. build the ovs
cd $OVS_DIR
./boot.sh
./configure --with-dpdk=$DPDK_BUILD
make
4. modify kernel paramters and reboot
[root@localhost ~]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/mapper/centos-root
ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16
rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet
nouveau.modeset=0 rd.driver.blacklist=nouveau default_hugepagesz=1G
hugepagesz=1G hugepages=8
5. install uio
modprobe uio
insmod $DPDK_BUILD/kmod/igb_uio.ko
6. bind the eth card
[root@localhost ~]# dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver
============================================
0000:15:00.1 'I350 Gigabit Network Connection' drv=igb_uio unused=
0000:15:00.3 'I350 Gigabit Network Connection' drv=igb_uio unused=
Network devices using kernel driver
===================================
0000:0b:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp11s0f0
drv=bnx2 unused=igb_uio
0000:0b:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp11s0f1
drv=bnx2 unused=igb_uio *Active*
0000:10:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp16s0f0
drv=bnx2 unused=igb_uio
0000:10:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp16s0f1
drv=bnx2 unused=igb_uio
0000:15:00.0 'I350 Gigabit Network Connection' if=ens2f0 drv=igb
unused=igb_uio
0000:15:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb
unused=igb_uio
|
|
|7. mount hugepage|
|
|
|mount -t hugetlbfs -o pagesize=1G none /dev/hugepages|
|
|
|
[root@localhost ~]# mount
...
none on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
8. setup and run db
$OVS_DIR/ovsdb/ovsdb-server
--remote=punix:/usr/local/var/run/openvswitch/db.sock
--remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile
--detach --log-file
|
9. start ovs-vswitchd
[root@localhost ~]# $OVS_DIR/vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4
--socket-mem 1024,0
2016-04-22T13:03:04Z|00001|dpdk|INFO|No -vhost_sock_dir provided -
defaulting to /usr/local/var/run/openvswitch
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 9 on socket 0
EAL: Detected lcore 3 as core 10 on socket 0
EAL: Detected lcore 4 as core 0 on socket 1
EAL: Detected lcore 5 as core 1 on socket 1
EAL: Detected lcore 6 as core 9 on socket 1
EAL: Detected lcore 7 as core 10 on socket 1
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 9 on socket 0
EAL: Detected lcore 11 as core 10 on socket 0
EAL: Detected lcore 12 as core 0 on socket 1
EAL: Detected lcore 13 as core 1 on socket 1
EAL: Detected lcore 14 as core 9 on socket 1
EAL: Detected lcore 15 as core 10 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x100000000 bytes
EAL: Virtual area found at 0x7fa9c0000000 (size = 0x100000000)
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Virtual area found at 0x7fa900000000 (size = 0x80000000)
EAL: Ask a virtual area of 0x40000000 bytes
EAL: Virtual area found at 0x7fa880000000 (size = 0x40000000)
EAL: Requesting 1 pages of size 1024MB from socket 0
EAL: TSC frequency is ~2400082 KHz
EAL: Master lcore 0 is ready (tid=55128c00;cpuset=[0])
EAL: PCI device 0000:15:00.0 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:15:00.1 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: PCI memory mapped at 0x7faa00000000
EAL: PCI memory mapped at 0x7faa00100000
PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521
EAL: PCI device 0000:15:00.2 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:15:00.3 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: PCI memory mapped at 0x7faa00104000
EAL: PCI memory mapped at 0x7faa00204000
PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521
Zone 0: name:<RG_MP_log_history>, phys:0x17fffdec0, len:0x2080,
virt:0x7fa9ffffdec0, socket_id:0, flags:0
Zone 1: name:<MP_log_history>, phys:0x17fd73d40, len:0x28a0c0,
virt:0x7fa9ffd73d40, socket_id:0, flags:0
Zone 2: name:<rte_eth_dev_data>, phys:0x17fd43380, len:0x2f700,
virt:0x7fa9ffd43380, socket_id:0, flags:0
2016-04-22T13:03:08Z|00002|ovs_numa|INFO|Discovered 8 CPU cores on
NUMA node 0
2016-04-22T13:03:08Z|00003|ovs_numa|INFO|Discovered 8 CPU cores on
NUMA node 1
2016-04-22T13:03:08Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and
16 CPU cores
2016-04-22T13:03:08Z|00005|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connecting...
2016-04-22T13:03:08Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connected
2016-04-22T13:03:08Z|00007|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports recirculation
2016-04-22T13:03:08Z|00008|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
label stack length probed as 3
2016-04-22T13:03:08Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports unique flow ids
2016-04-22T13:03:08Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_state
2016-04-22T13:03:08Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_zone
2016-04-22T13:03:08Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_mark
2016-04-22T13:03:08Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_label
VHOST_CONFIG: socket created, fd:32
VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user2
2016-04-22T13:03:08Z|00014|dpdk|INFO|Socket
/usr/local/var/run/openvswitch/vhost-user2 created for vhost-user port
vhost-user2
2016-04-22T13:03:08Z|00015|dpif_netdev|INFO|Created 1 pmd threads on
numa node 0
2016-04-22T13:03:08Z|00016|bridge|INFO|bridge br0: added interface
vhost-user2 on port 1
2016-04-22T13:03:08Z|00017|bridge|INFO|bridge br0: added interface br0
on port 65534
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da4940c0
hw_ring=0x7fa9da49c100 dma_addr=0x15a49c100
PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fa9da47ff40
hw_ring=0x7fa9da483f80 dma_addr=0x15a483f80
2016-04-22T13:03:08Z|00001|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'vhost-user2'
PMD: eth_igb_start(): <<
2016-04-22T13:03:08Z|00018|dpdk|INFO|Port 0: a0:36:9f:a1:dc:f9
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da4940c0
hw_ring=0x7fa9da49c100 dma_addr=0x15a49c100
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da467ec0
hw_ring=0x7fa9da46ff00 dma_addr=0x15a46ff00
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da44fe40
hw_ring=0x7fa9da457e80 dma_addr=0x15a457e80
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da437dc0
hw_ring=0x7fa9da43fe00 dma_addr=0x15a43fe00
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da41fd40
hw_ring=0x7fa9da427d80 dma_addr=0x15a427d80
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da407cc0
hw_ring=0x7fa9da40fd00 dma_addr=0x15a40fd00
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da3efc40
hw_ring=0x7fa9da3f7c80 dma_addr=0x15a3f7c80
PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance,
consider setting the TX WTHRESH value to 4, 8, or 16.
PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fa9da3d7bc0
hw_ring=0x7fa9da3dfc00 dma_addr=0x15a3dfc00
PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fa9da47ff40
hw_ring=0x7fa9da483f80 dma_addr=0x15a483f80
PMD: eth_igb_start(): <<
2016-04-22T13:03:08Z|00019|dpdk|INFO|Port 0: a0:36:9f:a1:dc:f9
2016-04-22T13:03:08Z|00002|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'vhost-user2'
2016-04-22T13:03:08Z|00003|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'dpdk0'
2016-04-22T13:03:08Z|00020|bridge|INFO|bridge br0: added interface
dpdk0 on port 2
VHOST_CONFIG: socket created, fd:43
VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user1
2016-04-22T13:03:08Z|00021|dpdk|INFO|Socket
/usr/local/var/run/openvswitch/vhost-user1 created for vhost-user port
vhost-user1
2016-04-22T13:03:08Z|00004|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'vhost-user2'
2016-04-22T13:03:08Z|00005|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'dpdk0'
2016-04-22T13:03:08Z|00006|dpif_netdev(pmd12)|INFO|Core 0 processing
port 'vhost-user1'
2016-04-22T13:03:08Z|00022|bridge|INFO|bridge br0: added interface
vhost-user1 on port 3
2016-04-22T13:03:08Z|00023|bridge|INFO|bridge br0: using datapath ID
0000a0369fa1dcf9
2016-04-22T13:03:08Z|00024|connmgr|INFO|br0: added service controller
"punix:/usr/local/var/run/openvswitch/br0.mgmt"
2016-04-22T13:03:08Z|00025|dpif_netdev|INFO|Created 1 pmd threads on
numa node 0
2016-04-22T13:03:08Z|00026|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.0
2016-04-22T13:03:08Z|00001|dpif_netdev(pmd20)|INFO|Core 0 processing
port 'vhost-user2'
2016-04-22T13:03:08Z|00002|dpif_netdev(pmd20)|INFO|Core 0 processing
port 'dpdk0'
2016-04-22T13:03:08Z|00003|dpif_netdev(pmd20)|INFO|Core 0 processing
port 'vhost-user1'
2016-04-22T13:03:15Z|00027|memory|INFO|17240 kB peak resident set size
after 10.2 seconds
2016-04-22T13:03:15Z|00028|memory|INFO|handlers:5 ports:4
revalidators:3 rules:5
10. create bridges and port
$OVS_DIR/utilities/ovs-vsctl add-br br0 -- set bridge br0
datapath_type=netdev
$OVS_DIR/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0
type=dpdk
ovs-vsctl add-port br0 vhost-user-2 -- set Interface vhost-user-2
type=dpdkvhostuser
ovs-vsctl add-port br0 vhost-user-2 -- set Interface vhost-user-2
type=dpdkvhostuser
11. run qemu
[root@localhost ~]# qemu-system-x86_64 -enable-kvm -name ubuntutest
-smp 4 -cpu host -m 1024 -hda /home/kvm/dpdknode1.qcow2 -vnc :19 -net
user,hostfwd=tcp::10022-:22 -net nic -chardev
socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user1
-netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device
virtio-net-pci,mac=52:54:00:00:00:01,netdev=mynet1 -object
memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages,share=on
-numa node,memdev=mem -mem-prealloc
qemu-system-x86_64: -netdev
type=vhost-user,id=mynet1,chardev=char1,vhostforce: chardev "char1"
went up
[root@localhost ~]# qemu-system-x86_64 --version
QEMU emulator version 2.4.1 (qemu-2.4.1-8.fc23), Copyright (c)
2003-2008 Fabrice Bellard
12. config IP of br0 and eth0 in VM
br0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel state
UNKNOWN qlen 500
link/ether a0:36:9f:a1:dc:f9 brd ff:ff:ff:ff:ff:ff
inet 192.168.201.2/24 brd 192.168.201.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::a236:9fff:fea1:dcf9/64 scope link
valid_lft forever preferred_lft forever
eth0 in VM is 192.168.201.3/24
13. ping br0 in VM
I can not ping, do tcpdump on br0
21:11:00.564650 ARP, Request who-has 192.168.201.2 tell 192.168.201.3,
length 28
21:11:00.564659 ARP, Reply 192.168.201.2 is-at a0:36:9f:a1:dc:f9,
length 28
br0 can get arp and response but VM dose not get the response
14. ovs-ofctl dump
[root@localhost ~]# ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000a0369fa1dcf9
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan
mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src
mod_tp_dst
1(vhost-user2): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
2(dpdk0): addr:a0:36:9f:a1:dc:f9
config: 0
state: 0
current: 1GB-FD
speed: 1000 Mbps now, 0 Mbps max
3(vhost-user1): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
LOCAL(br0): addr:a0:36:9f:a1:dc:f9
config: 0
state: 0
current: 10MB-FD COPPER
speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@localhost ~]# ovs-ofctl dump-ports br0
OFPST_PORT reply (xid=0x2): 4 ports
port 1: rx pkts=0, bytes=0, drop=?, errs=0, frame=?, over=?, crc=?
tx pkts=0, bytes=0, drop=202, errs=?, coll=?
port LOCAL: rx pkts=127, bytes=5646, drop=0, errs=0, frame=0,
over=0, crc=0
tx pkts=194, bytes=18000, drop=0, errs=0, coll=0
port 2: rx pkts=67, bytes=12354, drop=0, errs=0, frame=?, over=?, crc=?
tx pkts=135, bytes=8436, drop=0, errs=0, coll=?
* port 3: rx pkts=127, bytes=5646, drop=?, errs=0, frame=?, over=?,
crc=?*
* tx pkts=0, bytes=0, drop=194, errs=?, coll=?*
I found that vhost-user1 port, it can receive packages from VM, so rx
is not zero.
But it can not send packages to VM, so tx is zero, all packages are droped
So did you guys face this problem? how to fix that?
Thanks
Yours
Liu Chao
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
