I have ran into a ovs-vswitchd SEGV while trying to use the vhost-user feature with the latest dpdk. I have attempted to provide as much detail as I can.
versions compiled from source on CentOS 7.1 x86_64: - openvswitch 2.4.0 - http://openvswitch.org/releases/openvswitch-2.4.0.tar.gz - dpdk 2.1.0 - http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz - qemu - 2.4.0.1 - http://wiki.qemu-project.org/download/qemu-2.4.0.1.tar.bz2 Steps to reproduce: - Build and Install following these instructions http://wiki.qemu.org/Features/vhost-user-ovs-dpdk - setup details: hugepage config: GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=8 isolcpus=1,2,3,4,5,12,13,14,15,16,17,7,8,9,10,11,18,19,20,21,22,23 transparent_hugepage=never" # export DPDK_DIR=/root/dpdk/dpdk-2.1.0 # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/ # export DB_SOCK=/usr/local/var/run/openvswitch/db.sock # modprobe openvswitch # modprobe uio # insmod /root/dpdk/dpdk-2.1.0/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko # insmod /root/dpdk/dpdk-2.1.0/lib/librte_vhost/eventfd_link/eventfd_link.ko # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f0 # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f1 # /root/openvswitch/openvswitch-2.4.0/ovsdb/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach # rm -f /usr/local/var/run/openvswitch/vhost-user* # /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log # lspci | grep Ethernet 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser - Start the first VM like this: ./qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \ -display sdl \ -sdl \ -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-user1 \ -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01 \ -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages,share=on \ -numa node,memdev=mem -mem-prealloc \ -net user,hostfwd=tcp::10021-:22 \ -net nic \ /var/lib/libvirt/images/centos-vm1.qcow2 & - From another machine send iperf traffic to this VM. iperf -s (in the VM) iperf -c <ipaddr> -t 60 -P 12 (from another machine) - After about a few seconds, iperf stalls. network connectivity breaks. Can't ping the VM anymore. - Saw these in the ovs-vswitchd log files. Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA: Failed to allocate memory for mbuf. Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA: Failed to allocate memory for mbuf. Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA: Failed to allocate memory for mbuf. Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA: Failed to allocate memory for mbuf. Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA: Failed to allocate memory for mbuf. - shutdown iperf ... then shutdown the VM. - at this point ovs-vswitchd crashes with the following SEGV. [root@localhost vswitchd]# gdb ./ovs-vswitchd /var/spool/abrt/ccpp-2015-09-29-18\:47\:37-22012/coredump GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd...done. [New LWP 22051] [New LWP 22012] [New LWP 22013] [New LWP 22015] [New LWP 22039] [New LWP 22040] [New LWP 22041] [New LWP 22042] [New LWP 22043] [New LWP 22044] [New LWP 22045] [New LWP 22046] [New LWP 22047] [New LWP 22048] [New LWP 22049] [New LWP 22050] [New LWP 22014] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'. Program terminated with signal 11, Segmentation fault. #0 0x00007eff4d71df1f in rte_vhost_dequeue_burst () from /lib64/libdpdk.so Missing separate debuginfos, use: debuginfo-install glibc-2.17-78.el7.x86_64 (gdb) where #0 0x00007eff4d71df1f in rte_vhost_dequeue_burst () from /lib64/libdpdk.so #1 0x000000000056d103 in netdev_dpdk_vhost_rxq_recv (rxq_=0x7efe5a537600, packets=0x7eff4c9a2930, c=0x7eff4c9a292c) at lib/netdev-dpdk.c:952 #2 0x00000000004a2dec in netdev_rxq_recv (rx=0x7efe5a537600, buffers=0x7eff4c9a2930, cnt=0x7eff4c9a292c) at lib/netdev.c:652 #3 0x0000000000478f33 in dp_netdev_process_rxq_port (pmd=0x231c310, port=0x2166990, rxq=0x7efe5a537600) at lib/dpif-netdev.c:2520 #4 0x0000000000479648 in pmd_thread_main (f_=0x231c310) at lib/dpif-netdev.c:2680 #5 0x00000000004f60f3 in ovsthread_wrapper (aux_=0x231c150) at lib/ovs-thread.c:340 #6 0x00007eff4d276df5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007eff4ca9a1ad in clone () from /lib64/libc.so.6
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
