On Tue, May 10, 2016 at 6:31 PM, Jesse Gross <je...@kernel.org> wrote: > On Tue, May 10, 2016 at 3:26 AM, Chandran, Sugesh > <sugesh.chand...@intel.com> wrote: >>> -----Original Message----- >>> From: Jesse Gross [mailto:je...@kernel.org] >>> Sent: Friday, May 6, 2016 5:00 PM >>> To: Chandran, Sugesh <sugesh.chand...@intel.com> >>> Cc: pravin shelar <pshe...@ovn.org>; ovs dev <dev@openvswitch.org> >>> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling >>> performance using DPDK Rx checksum offloading feature. >>> >>> On Fri, May 6, 2016 at 1:13 AM, Chandran, Sugesh >>> <sugesh.chand...@intel.com> wrote: >>> >> -----Original Message----- >>> >> From: Jesse Gross [mailto:je...@kernel.org] >>> >> Sent: Friday, May 6, 2016 1:58 AM >>> >> To: Chandran, Sugesh <sugesh.chand...@intel.com> >>> >> Cc: pravin shelar <pshe...@ovn.org>; ovs dev <dev@openvswitch.org> >>> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling >>> >> performance using DPDK Rx checksum offloading feature. >>> >> >>> >> On Thu, May 5, 2016 at 1:26 AM, Chandran, Sugesh >>> >> <sugesh.chand...@intel.com> wrote: >>> >> >> -----Original Message----- >>> >> >> From: Jesse Gross [mailto:je...@kernel.org] >>> >> >> Sent: Wednesday, May 4, 2016 10:06 PM >>> >> >> To: Chandran, Sugesh <sugesh.chand...@intel.com> >>> >> >> Cc: pravin shelar <pshe...@ovn.org>; ovs dev >>> <dev@openvswitch.org> >>> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling >>> >> >> performance using DPDK Rx checksum offloading feature. >>> >> >> >>> >> >> On Wed, May 4, 2016 at 8:58 AM, Chandran, Sugesh >>> >> >> <sugesh.chand...@intel.com> wrote: >>> >> >> >> -----Original Message----- >>> >> >> >> From: Jesse Gross [mailto:je...@kernel.org] >>> >> >> >> Sent: Thursday, April 28, 2016 4:41 PM >>> >> >> >> To: Chandran, Sugesh <sugesh.chand...@intel.com> >>> >> >> >> Cc: pravin shelar <pshe...@ovn.org>; ovs dev >>> >> <dev@openvswitch.org> >>> >> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving >>> >> >> >> tunneling performance using DPDK Rx checksum offloading feature. >>> >> >> > >>> >> >> >> That sounds great, thanks for following up. In the meantime, do >>> >> >> >> you have any plans for transmit side checksum offloading? >>> >> >> > [Sugesh] The vectorization on Tx side is getting disabled when >>> >> >> > DPDK Tx >>> >> >> checksum offload is enabled. This causes performance drop in OVS. >>> >> >> > However We don’t find any such impact when enabling Rx checksum >>> >> >> offloading(though this disables Rx vectorization). >>> >> >> >>> >> >> OK, I see. Does the drop in throughput cause performance to go >>> >> >> below the baseline even for UDP tunnels with checksum traffic? (I >>> >> >> guess small and large packets might have different results here.) >>> >> >> Or is it that it reduce performance for unrelated traffic? If it's >>> >> >> the latter case, can we find a way to use offloading conditionally? >>> >> > [Sugesh] We tested for 64 byte UDP packet stream and found that the >>> >> > performance is better when the offloading is turned off. This is >>> >> > for any >>> >> traffic through the port. >>> >> > DPDK doesn’t support conditional offloading for now. >>> >> > In other words DPDK can't do selective vector packet processing on a >>> port. >>> >> > As far as I know there are some technical difficulties to enable >>> >> > offload + vectorization together in DPDK. >>> >> >>> >> My guess is the results might be different for larger packets since >>> >> those cases will stress checksumming more and rx/tx routines less. >>> >> >>> >> In any case, I think this is an area that is worthwhile to continue >>> investigating. >>> >> My expectation is that tunneled packets with outer UDP checksums will >>> >> be a use case that is hit increasingly frequently with OVS DPDK - for >>> >> example, OVN will likely start exercising this soon. >>> > [Sugesh]Totally agreed, I will do PHY-PHY, PHY-TUNNEL-PHY tests with >>> > different size traffic >>> > streams(64 Byte, 512, 1024, 1500) when checksum enabled/disabled and >>> see the impact. >>> > Is there any other traffic pattern/tests that we have to consider? >>> >>> I think that should cover it pretty well. Thanks a lot! >> [Sugesh] Please find below for the test results in different scenarios. >> >> Native(Rx, Tx checksum offloading OFF) >> Test 64 Bytes 128 Bytes 256 Bytes 512 >> Bytes 1500 bytes Mix >> PHY-PHY-BIDIR 9.2 8.445 4.528 >> 2.349 0.822 6.205 >> PHY-VM-PHY-BIDIR 2.564 2.503 2.205 >> 1.901 0.822 2.29 >> PHY-VXLAN-PHY 4.165 4.084 3.834 2.147 >> 0.849 3.964 >> >> >> Rx Checksum ON/Rx Vector OFF >> >> Test 64 Bytes 128 Bytes 256 Bytes 512 >> Bytes 1500 bytes Mix >> PHY-PHY-BIDIR 9.12 8.445 4.528 >> 2.349 0.822 6.205 >> PHY-VM-PHY-BIDIR 2.535 2.513 2.21 >> 1.913 0.822 2.25 >> PHY-VXLAN-PHY 4.475 4.47 3.834 2.147 >> 0.849 4.4 >> >> >> >> Tx Checksum ON/Tx Vector OFF >> >> Test 64 Bytes 128 Bytes 256 Bytes 512 >> Bytes 1500 bytes Mix >> PHY-PHY-BIDIR 7.89 7.935 4.528 >> 2.349 0.822 6.205 >> PHY-VM-PHY-BIDIR 2.455 2.401 2.118 >> 1.855 0.822 2.2 >> PHY-VXLAN-PHY 3.347 3.301 3.185 2.147 >> 0.849 3.218 >> >> >> >> Rx, Tx Checksum ON/Rx, Tx Vector OFF >> Test 64 Bytes 128 Bytes 256 Bytes 512 >> Bytes 1500 bytes Mix >> PHY-PHY-BIDIR 7.75 7.74 4.528 >> 2.349 0.822 6.205 >> PHY-VM-PHY-BIDIR 2.44 2.401 2.141 >> 1.862 0.822 2.21 >> PHY-VXLAN-PHY 3.53 3.53 3.53 2.147 >> 0.849 3.52 >> >> Findings:- >> 1) For bigger packet sizes, (<= 256 bytes) the performance numbers are >> almost the same in all the scenarios. >> 2) Tx checksum offload doesn’t give any performance improvement in all the >> case. In fact in some scenarios, it shows low performance compared to native. >> 3) Rx checksum offload provide tunneling performance improvement mainly on >> small sized packets. >> >> Please let me know your comments/thoughts on this? > > Thanks, this is great data. > > I agree that it doesn't make any sense to pursue transmit checksum > offload in OVS at this point since it doesn't result in improvements > in any case. I'm a bit surprised and it would be interesting to > understand in more detail what is going on and if anything can be > improved to allow both checksum offloading and vectoring at the same > time but I guess that's more of a DPDK question than one for OVS. > > I'm a little bit torn as to whether we should apply your rx checksum > offload patch in the meantime while we wait for DPDK to offer the new > API. It looks like we'll have a 10% gain with tunneling in exchange > for a 1% loss in other situations, so the call obviously depends on > use case. Pravin, Daniele, others, any opinions? > There could be a way around the API issue and avoid the 1% loss. netdev API could be changed to set packet->mbuf.ol_flags to (PKT_RX_IP_CKSUM_BAD | PKT_RX_L4_CKSUM_BAD) if the netdev implementation does not support rx checksum offload. Then there is no need to check the rx checksum flags in dpif-netdev. And the checksum can be directly checked in tunneling code where we actually need to. Is there any issue with this approach?
> Do you happen to know in the case of tunneling how the best case > (which looks like with rx checksums on) of outer UDP checksums > compares to tunneling without outer checksums (which would presumably > allow us to keep vectoring)? I should have mentioned this earlier, so > I'm more just curious if you have a ballpark figure. _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
