Hi Daniele, You are right – after pulling the latest master branch, the ovs-vswitchd doesn’t crash any more. Whether it’s b59cc14e032da370021794bfd1bdd3d67e88a9a3 (netdev-dpdk: Use instant sending instead of queueing of packets.) or one of the other queue-related patches, I can’t say. There were a bunch of updates since last week ;-)
A quick repetition of some of our performance measurements shows that the performance is slightly better for multi-queue than for single queue and a little bit higher than before with using the older ovs-version. But that needs to be proofed.. The configured_tx_queues for dpdk ports is still 21. I saw another patch which introduces n_txq which I haven’t tried yet but will do soon…. Thank you very much! BR/Mechthild From: Daniele Di Proietto [mailto:diproiet...@ovn.org] Sent: Wednesday, July 13, 2016 3:40 AM To: Mechthild Buescher Cc: Aaron Conole; b...@openvswitch.org Subject: Re: [ovs-discuss] bug in ovs-vswitchd?! Hi Mechthild, Have you tried with the latest master? I suspect this might have already been fixed by b59cc14e032d("netdev-dpdk: Use instant sending instead of queueing of packets."). In this case I need to backport that on branch-2.5 as well. Thanks, Daniele 2016-07-12 17:23 GMT-07:00 Mechthild Buescher <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>>: Hi Aaron, I checked whether vhost-sock-permissions is needed to run our tests and can confirm that it's not needed. The removal of vhost-sock-permissions, however, does not make a difference for 2 cores and 1 queue - ovs-vswitchd is still crashing. Regarding the solution for the socket access: It was needed to updated some libvirt related apparmor files to get it running (e.g. in the dedicated /etc/apparmor.d/libvirt/libvirt-<uuid>.files). As far as I remember, for older libvirt version, it was also needed to update /etc/qemu.conf. But the updates very much depend on which libvirt version is used and where the sockets are stored. So I cannot give a general solution to socket-access problem as there might be different reason. Of course qemu must be client of the socket if ovs is the server - I guess you know this ;-) I failed to configure less than 21 tx queues, neither ovs-vsctl --no-wait set Open_vSwitch . other_config:n-dpdk-txqs=2 nor ovs-vsctl --no-wait set Interface dpdk0 options:n_txq=2 changes the value of dpdk0 1/2: (dpdk: configured_rx_queues=2, configured_tx_queues=21, requested_rx_queues=2, requested_tx_queues=21) But since the number of tx_queues is 21 for all our configurations, I wonder why it leads to a crash of ovs-vswitchd only if the number of cores is 2 and the number of queues is 1. Best regards, Mechthild -----Original Message----- From: Aaron Conole [mailto:acon...@redhat.com<mailto:acon...@redhat.com>] Sent: Tuesday, July 12, 2016 4:46 PM To: Mechthild Buescher Cc: Stokes, Ian; b...@openvswitch.org<mailto:b...@openvswitch.org> Subject: Re: [ovs-discuss] bug in ovs-vswitchd?! Mechthild Buescher <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>> writes: > Hi Aaron, > > I think that the vhost-sock-permissions is not needed - I will check > whether it makes a different. It's a left-over from an earlier > configuration where we had the problem that qemu (started by libvirt) > wasn't able to access the socket. This problem has been solved. Can you share the solution? It's unrelated, but this is something I'm trying to solve at the moment. > The tx queues are not configured by us, so I don't know where this > value comes from. Maybe it's the default value?! So, it's not > intended. Okay, that could likely be the problem. Please try setting the TX queues first, and if that resolves the crash there is either a setup script or possibly code error. > cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 62 > model name : Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz Thanks for this, it helps to locate which vectorization code dpdk will use. > stepping : 4 > microcode : 0x40c > cpu MHz : 1200.000 > cache size : 25600 KB > physical id : 0 > siblings : 20 > core id : 0 > cpu cores : 10 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 13 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov > pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx > pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl > xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor > ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 > x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida > arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid > fsgsbase smep erms > bogomips : 4799.93 > clflush size : 64 > cache_alignment : 64 > address sizes : 46 bits physical, 48 bits virtual > power management: > > Thanks again, > > BR/Mechthild > > -----Original Message----- > From: Aaron Conole [mailto:acon...@redhat.com<mailto:acon...@redhat.com>] > Sent: Monday, July 11, 2016 11:06 PM > To: Mechthild Buescher > Cc: Stokes, Ian; b...@openvswitch.org<mailto:b...@openvswitch.org> > Subject: Re: [ovs-discuss] bug in ovs-vswitchd?! > > Mechthild Buescher > <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>> > writes: > >> Hi Aaron, >> >> Sorry for being unclear regarding the VM: I meant the DPDK usage >> inside the VM. So, the fault happens when using the VM. Inside the VM >> I can either bind the interfaces to DPDK or to linux - in both cases, >> the fault occurs. >> >> And I haven't applied any patch. I used the latest available version >> from the master branch - I don't know whether any patch is upstreamed >> to the master branch. > > Okay - I wonder what the vhost-sock-permissions command line is all about, > then? > > Can you confirm that 21 tx queues is not intended, then (21 tx queues >> is showing in your configuration output)? > > Also, please send the cpu information (cat /proc/cpuinfo on the host). > >> Thanks in advance for your help, >> >> BR/Mechthild >> >> -----Original Message----- >> From: Aaron Conole [mailto:acon...@redhat.com<mailto:acon...@redhat.com>] >> Sent: Monday, July 11, 2016 7:22 PM >> To: Mechthild Buescher >> Cc: Stokes, Ian; b...@openvswitch.org<mailto:b...@openvswitch.org> >> Subject: Re: [ovs-discuss] bug in ovs-vswitchd?! >> >> Mechthild Buescher >> <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>> >> writes: >> >>> Hi Ian, >>> >>> Thanks for the fast reply! I also did some further investigations >>> where I could see that ovs-vswitchd usually keeps alive for >>> receiving packets but crashes for sending packets. >>> >>> Regarding your question: >>> 1. We are running 1 VM with 2 vhost ports (in the simplified setup; >>> in the complete setup we use 1 VM & 5 vhost ports) >>> >>> 2. We are using libvirt to start the VM which is configured to use qemu: >>> /usr/bin/qemu-system-x86_64 -name >>> guest=ubuntu11_try,debug-threads=on >>> -S -machine pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 8192 >>> -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -object >>> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/huge_1G/ >>> l >>> i >>> bvirt/qemu,share=yes,size=8589934592 >>> -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 -uuid >>> 8a2ad7a3-9da1-4c69-a2ff-c7a680d9bc4a -no-user-config -nodefaults >>> -chardev >>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-300-ubuntu11 >>> _ >>> t >>> ry/monitor.sock,server,nowait -mon >>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc >>> -no-shutdown -boot strict=on -device >>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive >>> file=/root/perf/vms/ubuntu11.qcow2,format=qcow2,if=none,id=drive-vir >>> t >>> i >>> o-disk0 >>> -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0, >>> i >>> d >>> =virtio-disk0,bootindex=1 >>> -netdev tap,fd=21,id=hostnet0 -device >>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3c:92:47,bus=pci. >>> 0 >>> ,addr=0x3 >>> -netdev tap,fd=23,id=hostnet1 -device >>> virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:3c:a3:47,bus=pci. >>> 0 >>> ,addr=0x4 -chardev >>> socket,id=charnet2,path=/var/run/openvswitch/vhost111 -netdev >>> type=vhost-user,id=hostnet2,chardev=charnet2 -device >>> virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:a0:11:02,bus=pci. >>> 0 >>> ,addr=0x5 -chardev >>> socket,id=charnet3,path=/var/run/openvswitch/vhost112 -netdev >>> type=vhost-user,id=hostnet3,chardev=charnet3 -device >>> virtio-net-pci,netdev=hostnet3,id=net3,mac=52:54:00:a0:11:03,bus=pci. >>> 0 >>> ,addr=0x6 >>> -chardev pty,id=charserial0 -device >>> isa-serial,chardev=charserial0,id=serial0 -vnc >>> 127.0.0.1:0<http://127.0.0.1:0> -device >>> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device >>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on >>> >>> 3. In the VM we have both kind of interface bindings, virtio-pci and >>> igb_uio. For both type of interfaces, the crash of ovs-vswitchd can >>> be observed (The VM is still alive). >>> >>> 4. The ovs-vswitchd is started as follows and is configured to use >>> vxlan tunnel: >>> ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true >>> ovs-vsctl --no-wait set Open_vSwitch . >>> other_config:dpdk-lcore-mask=0x1 ovs-vsctl --no-wait set >>> Open_vSwitch . other_config:dpdk-socket-mem=4096,0 >>> ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6 >>> ovs-vsctl --no-wait set Interface dpdk0 options:n_rxq=2 ovs-vsctl >>> --no-wait set Interface vhost111 options:n_rxq=1 ovs-vsctl --no-wait >>> set Interface vhost112 options:n_rxq=1 ovs-vsctl --no-wait set >>> Open_vSwitch . other_config:vhost-sock-permissions=766 >> >> Have you, perchance, applied some extra patches? This was proposed, >>> but not accepted, as a possible workaround for a permissions issue >>> with ovs dpdk. >> >>> ovs-vswitchd --pidfile=$DB_PID --detach --monitor >>> --log-file=$LOG_FILE -vfile:dbg --no-chdir -vconsole:emer --mlockall >>> unix:$DB_SOCK >>> >>> ovs-vsctl add-port br-int vhost111 -- set Interface vhost111 >>> type=dpdkvhostuser ofport_request=11 ovs-vsctl add-port br-int >>> vhost112 -- set Interface vhost112 type=dpdkvhostuser >>> ofport_request=12 ovs-vsctl add-br br-int -- set bridge br-int >>> datapath_type=netdev ovs-vsctl set Bridge br-int >>> other_config:datapath-id=0000f2b811144f41 >>> ovs-vsctl set Bridge br-int protocols=OpenFlow13 ovs-vsctl add-port >>> br-int vxlan0 -- set interface vxlan0 type=vxlan >>> options:remote_ip=10.1.2.2 options:key=flow ofport_request=100 >>> >>> 5. The ovs-log is attached - it contains the log from start to crash >>> (with debug information). The crash has been provoked by setting up >>> an virtio-pci in the VM, so no DPDK is used in the VM for this scenario. >>> >>> 6. The DPDK versions are: >>> Host: dpdk 16.04 latest commit >>> b3b9719f18ee83773c6ed7adda300c5ac63c37e9 >>> VM: (not used in this scenario) dpdk 2.2.0 >> >> For confirmation, this happens whether or not you use a VM? I just >>> want to make sure. It's usually best to pair dpdk versions whenever >>> possible. >> >>> BR/Mechthild >>> >>> -----Original Message----- >>> From: Stokes, Ian [mailto:ian.sto...@intel.com<mailto:ian.sto...@intel.com>] >>> Sent: Thursday, July 07, 2016 1:57 PM >>> To: Mechthild Buescher; b...@openvswitch.org<mailto:b...@openvswitch.org> >>> Subject: RE: bug in ovs-vswitchd?! >>> >>> Hi Mechthild, >>> >>> I've tried to reproduce this issue on my setup (Fedora 22 kerenl >>> 4.1.8) but have not been able to reproduce it. >>> >>> A few questions to help the investigation >>> >>> 1. Are you running 1 or 2 VMs in the setup (i.e. 1 vm with 2 vhost >>> user ports or 2 vms with 1 vhost user port each?) 2. What are the >>> parameters being used to launch the VM/s attached to the vhost user >>> ports? >>> 3. Inside the VM, are the interfaces bound to? igb_uio (i.e. using >>> dpdk app inside the guest) or are the interfaces being used as >>> kernel devices inside the VM? >>> 4. What parameters are you launching OVS with? >>> 5. Can you provide an ovs log? >>> 6. Can you confirm the DPDK version you are using in the host/VM (If >>> being used in the VM). >>> >>> Thanks >>> Ian >>> >>>> From: discuss >>>> [mailto:discuss-boun...@openvswitch.org<mailto:discuss-boun...@openvswitch.org>] >>>> On Behalf Of >>>> Mechthild Buescher >>>> Sent: Wednesday, July 06, 2016 1:54 PM >>>> To: b...@openvswitch.org<mailto:b...@openvswitch.org> >>>> Subject: [ovs-discuss] bug in ovs-vswitchd?! >>>> >>>> Hi all, >>>> >>>> we are using ovs with dpdk-interfaces and vhostuser-interfaces and >>>> want to try the VMs with different multi-queue settings. When we >>>> specify 2 cores and 2 multi-queues for a dpdk-interface but only >>>> one queue >for the vhost-interfaces, ovs-vswitchd crashes at start >>>> of the VM (or latest when traffic is sent). >>>> >>>> The version of ovs is : 2.5.90, master branch, latest commit >>>> 7a15be69b00fe8f66a3f3929434b39676f325a7a) >>>> It has been built and is running on: Linux version >>>> 3.13.0-87-generic >>>> (buildd@lgw01-25) (gcc version 4.8.4 (Ubuntu >>>> 4.8.4-2ubuntu1~14.04.3) >>>> ) #133-Ubuntu SMP Tue May 24 18:32:09 UTC 2016 >>>> >>>> The configuration is: >>>> ovs-vsctl show >>>> 0e191ed4-040b-458c-bad8-feb6f7c90e3a >>>> Bridge br-prv >>>> Port br-prv >>>> Interface br-prv >>>> type: internal >>>> Port "dpdk0" >>>> Interface "dpdk0" >>>> type: dpdk >>>> options: {n_rxq="2"} >>>> Bridge br-int >>>> Port br-int >>>> Interface br-int >>>> type: internal >>>> Port "vhost112" >>>> Interface "vhost112" >>>> type: dpdkvhostuser >>>> options: {n_rxq="1"} >>>> Port "vhost111" >>>> Interface "vhost111" >>>> type: dpdkvhostuser >>>> options: {n_rxq="1"} >>>> Port "vxlan0" >>>> Interface "vxlan0" >>>> type: vxlan >>>> options: {key=flow, remote_ip="10.1.2.2"} >>>> >>>> ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 0 core_id >>>>1: >>>> port: vhost112 queue-id: 0 >>>> port: dpdk0 queue-id: 1 pmd thread numa_id >>>>0 core_id 2: >>>> port: dpdk0 queue-id: 0 >>>> port: vhost111 queue-id: 0 >>>> >>>> appctl-m dpif/show >>>> br-int: >>>> br-int 65534/6: (tap) >>>> vhost111 11/3: (dpdkvhostuser: >>>>configured_rx_queues=1, configured_tx_queues=1, >>>>requested_rx_queues=1, >>>>requested_tx_queues=21) >>>> vhost112 12/5: (dpdkvhostuser: >>>>configured_rx_queues=1, configured_tx_queues=1, >>>>requested_rx_queues=1, >>>>requested_tx_queues=21) >>>> vxlan0 100/4: (vxlan: key=flow, >>>>remote_ip=10.1.2.2) >>>> br-prv: >>>> br-prv 65534/1: (tap) >>>> dpdk0 1/2: (dpdk: >>>>configured_rx_queues=2, configured_tx_queues=21, >>>>requested_rx_queues=2, >>>>requested_tx_queues=21) >> >> I'm a little concerned about the numbers reported here. 21 tx queues >> is a bit much, I think. I haven't tried reproducing this yet, but >> can you confirm this is desired? >> >>>> >>>> (gdb) bt >>>> #0 0x00000000005356e4 in ixgbe_xmit_pkts_vec () >>>> #1 0x00000000006df384 in rte_eth_tx_burst (nb_pkts=<optimized >>>> out>, tx_pkts=<optimized out>, queue_id=1, port_id=<optimized out>) >>>> at >>>> /opt/dpdk-16.04/x86_64-native-linuxapp-gcc//include/rte_ethdev.h:27 >>>> 9 >>>> 1 >>>> #2 dpdk_queue_flush__ (qid=<optimized out>, dev=<optimized out>) >>>> at >>>> lib/netdev-dpdk.c:1099 >>>> #3 dpdk_queue_flush (qid=<optimized out>, dev=<optimized out>) at >>>> lib/netdev-dpdk.c:1133 >>>> #4 netdev_dpdk_rxq_recv (rxq=0x7fbe127ad4c0, >>>> packets=0x7fc26761e408, >>>> c=0x7fc26761e400) at lib/netdev-dpdk.c:1312 >>>> #5 0x000000000061be98 in netdev_rxq_recv (rx=<optimized out>, >>>> batch=batch@entry=0x7fc26761e400) at lib/netdev.c:628 >>>> #6 0x00000000005f17bb in dp_netdev_process_rxq_port >>>> (pmd=pmd@entry=0x29ea810, rxq=<optimized out>, port=<optimized >>>> out>, port=<optimized out>) >>>> at lib/dpif-netdev.c:2619 >>>> #7 0x00000000005f1b27 in pmd_thread_main (f_=0x29ea810) at >>>> lib/dpif-netdev.c:2864 >>>> #8 0x000000000067dde4 in ovsthread_wrapper (aux_=<optimized out>) >>>> at >>>> lib/ovs-thread.c:342 >>>> #9 0x00007fc26b90e184 in start_thread (arg=0x7fc26761f700) at >>>> pthread_create.c:312 >>>> #10 0x00007fc26af2237d in clone () at >>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 >>>> >>>> This is the minimal configuration which leads to the fault. Our >>>> complete configuration contains more vhostuser interfaces than >>>> above. We observed that only the combination of 2 cores/queues for >>>> dpdk- interface and 1 queue for vhostuser interfaces results in an >>>> ovs-vswitchd crash, in detail: >>>> Dpdk0: 1 cores/queues & all vhost-ports: 1 queue => successful >>>> Dpdk0: 2 cores/queues & all vhost-ports: 1 queue => crash >>>> Dpdk0: 2 cores/queues & all vhost-ports: 2 queue => successful >>>> Dpdk0: 4 cores/queues & all vhost-ports: 1 queue => successful >>>> Dpdk0: 4 cores/queues & all vhost-ports: 2 queue => successful >>>> Dpdk0: 4 cores/queues & all vhost-ports: 4 queue => successful >>>> >>>> Do you have any suggestions? >> >> Can you please also supply the cpu (model number) that you're using? >> >> Thanks, >> Aaron >> >>>> Best regards, >>>> >>>> Mechthild Buescher _______________________________________________ discuss mailing list discuss@openvswitch.org<mailto:discuss@openvswitch.org> http://openvswitch.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
