Hi, Recently we upgraded to ovs 2.5 and use the datapath in the ovs code(means we do not use kernel datapath), however, when we test vxlan performance between two VMs on different compute node, we observed the bandwidth can only achieve 5Gbps on the 10Gbps intel nic pair, and we are still using kernel v3.14.57.
We found lots of following warning (see stack in the PS) in dmesg and thinks it causes the bad performance. After trying to read the source code of the kernel and ovs 2.5 datapath code, it seems gso_type 513(0x201) cause the problem, since our VM tap interface does not have tx-udp_tnl-segmentation(NETIF_F_GSO_UDP_TUNNEL_BIT) flag enabled by default, net_gso_ok return false which cause __skb_gso_segment being called, and since tcpv4_gso_complete have set the skb->ip_summed to CHECKSUM_UNCESSARY, which cause skb_warn_bad_offload being called. My wonder is whether this is a desired behavior, or OVS should clear the NETIF_F_GSO_UDP_TUNNEL_BIT bit when send the inner packet (may be GSOed) to tap interface on the host? Thanks a lot. PS: [1911068.644732] WARNING: CPU: 2 PID: 0 at net/core/dev.c:2246 skb_warn_bad_offload+0xc8/0xd3() [1911068.644735] : caps=(0x00000020001948c9, 0x0000000000000000) len=1597 data_len=1531 gso_size=1348 gso_type=513 ip_summed=1 [1911068.644736] Modules linked in: vp_toa(O) vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 xt_nat xt_REDIRECT nls_utf8 nls_cp437 vfat fat ipmi_devintf vhost_net vhost macvtap macvlan tun loop nbd xt_comment xt_state tcp_diag inet_diag gre vxlan libcrc32c xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge iscsi_trgt(O) crc32c ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc 8021q garp stp mrp llc nbsramdisk(O) snd_pcm snd_timer snd soundcore x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel evdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd mgag200 ttm drm_kms_helper drm shpchp iTCO_wdt iTCO_vendor_support lpc_ich ipmi_si mei_me ipmi_msghandler mei dcdbas pcspkr ehci_pci ehci_hcd usbcore usb_common mfd_core wmi acpi_power_meter processor thermal_sys button ext4 crc16 mbcache jbd2 dm_mod md_mod sg sd_mod crc_t10dif crct10dif_common igb i2c_algo_bit ixgbe i2c_core dca megaraid_sas ptp pps_core scsi_mod mdio [last unloaded: openvswitch] [1911068.644827] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W O 3.14.57-openstack-amd64 #1 [1911068.644829] Hardware name: Dell Inc. PowerEdge R730xd/0H21J3, BIOS 1.3.6 06/03/2015 [1911068.644830] 0000000000000000 0000000000000009 ffffffff81524cd0 ffff88307f423558 [1911068.644837] ffffffff8106940b ffff882f84279940 ffff883fb1d14000 0000000000000201 [1911068.644842] 0000000000000001 0000000000000010 ffffffff810694fa ffffffff81755988 [1911068.644847] Call Trace: [1911068.644848] <IRQ> [<ffffffff81524cd0>] ? dump_stack+0x41/0x51 [1911068.644857] [<ffffffff8106940b>] ? warn_slowpath_common+0x8b/0xc0 [1911068.644860] [<ffffffff810694fa>] ? warn_slowpath_fmt+0x4a/0x50 [1911068.644865] [<ffffffff812c5cc8>] ? ___ratelimit+0xb8/0x160 [1911068.644869] [<ffffffff8152632d>] ? skb_warn_bad_offload+0xc8/0xd3 [1911068.644873] [<ffffffff810ae354>] ? __wake_up_common+0x54/0x90 [1911068.644878] [<ffffffff814354a9>] ? __skb_gso_segment+0x79/0xc0 [1911068.644881] [<ffffffff81435695>] ? dev_hard_start_xmit+0x1a5/0x580 [1911068.644887] [<ffffffff81456045>] ? sch_direct_xmit+0x105/0x1d0 [1911068.644890] [<ffffffff81435c7a>] ? __dev_queue_xmit+0x20a/0x4d0 [1911068.644896] [<ffffffffa06e9387>] ? do_output.isra.35+0x47/0x140 [openvswitch] [1911068.644900] [<ffffffffa06ea24a>] ? do_execute_actions+0xdca/0x11d0 [openvswitch] [1911068.644904] [<ffffffff8142d0df>] ? skb_flow_dissect+0x18f/0x3b0 [1911068.644909] [<ffffffffa06f5f22>] ? flow_lookup.isra.6+0x42/0xc0 [openvswitch] [1911068.644914] [<ffffffffa06ea68b>] ? ovs_execute_actions+0x3b/0x120 [openvswitch] [1911068.644918] [<ffffffffa06eeea9>] ? ovs_dp_process_packet+0x99/0x130 [openvswitch] [1911068.644923] [<ffffffffa06efc7e>] ? key_extract+0x86e/0xc60 [openvswitch] [1911068.644928] [<ffffffffa06f7335>] ? ovs_vport_receive+0x85/0xf0 [openvswitch] [1911068.644933] [<ffffffffa06efb0e>] ? key_extract+0x6fe/0xc60 [openvswitch] [1911068.644937] [<ffffffffa06f7335>] ? ovs_vport_receive+0x85/0xf0 [openvswitch] [1911068.644943] [<ffffffff8106e804>] ? __local_bh_enable_ip+0x74/0xe0 [1911068.644946] [<ffffffff81435cc0>] ? __dev_queue_xmit+0x250/0x4d0 [1911068.644950] [<ffffffff8106e804>] ? __local_bh_enable_ip+0x74/0xe0 [1911068.644953] [<ffffffff8106e804>] ? __local_bh_enable_ip+0x74/0xe0 [1911068.644959] [<ffffffffa07076f0>] ? vxlan_rcv+0x290/0x6d0 [openvswitch] [1911068.644964] [<ffffffffa05f08ad>] ? udp_packet+0x7d/0x90 [nf_conntrack] [1911068.644969] [<ffffffffa05ea37d>] ? nf_conntrack_in+0x26d/0x920 [nf_conntrack] [1911068.644975] [<ffffffffa0707cab>] ? vxlan_udp_encap_recv+0x17b/0x220 [openvswitch] [1911068.644981] [<ffffffff81499a40>] ? udp_queue_rcv_skb+0x140/0x340 [1911068.644984] [<ffffffff8149a41c>] ? __udp4_lib_rcv+0x37c/0x6c0 [1911068.644989] [<ffffffff8146b90e>] ? ip_local_deliver_finish+0x9e/0x200 [1911068.644992] [<ffffffff814333cb>] ? __netif_receive_skb_core+0x6db/0x830 [1911068.644996] [<ffffffff8101de25>] ? read_tsc+0x5/0x20 [1911068.645000] [<ffffffff8143375f>] ? netif_receive_skb_internal+0x1f/0x80 [1911068.645004] [<ffffffff81433f43>] ? dev_gro_receive+0x2c3/0x480 [1911068.645007] [<ffffffff8143423f>] ? napi_gro_receive+0x1f/0xf0 [1911068.645018] [<ffffffffa00e4a5d>] ? ixgbe_clean_rx_irq+0x81d/0xb70 [ixgbe] [1911068.645026] [<ffffffffa00e5d40>] ? ixgbe_poll+0x4f0/0x9a0 [ixgbe] [1911068.645031] [<ffffffff810c21c1>] ? handle_irq_event+0x51/0x70 [1911068.645035] [<ffffffff812cd92a>] ? trace_hardirqs_off_thunk+0x3a/0x6c [1911068.645039] [<ffffffff8152aabc>] ? _raw_spin_lock_irqsave+0x2c/0x70 [1911068.645043] [<ffffffff81433b62>] ? net_rx_action+0x152/0x270 [1911068.645047] [<ffffffff8106e4ff>] ? __do_softirq+0x12f/0x2e0 [1911068.645051] [<ffffffff8106e9a5>] ? irq_exit+0xb5/0xc0 [1911068.645056] [<ffffffff81535ab6>] ? do_IRQ+0x66/0x110 [1911068.645059] [<ffffffff8152af72>] ? common_interrupt+0x72/0x72 [1911068.645060] <EOI> [<ffffffff8152afe4>] ? retint_restore_args+0x13/0x13 [1911068.645067] [<ffffffff810c12d9>] ? cpu_startup_entry+0x159/0x250 [1911068.645070] [<ffffffff810c12b5>] ? cpu_startup_entry+0x135/0x250 [1911068.645072] --[ end trace 80f530a2f3ff3740 ]--
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
