Inline. > -----Original Message----- > From: Tom Herbert [mailto:[email protected]] > Sent: Tuesday, December 9, 2014 9:31 PM > To: Pankaj Garg > Cc: Darrel Lewis (darlewis); [email protected] > Subject: Re: [nvo3] New Version Notification for draft-herbert-vxlan-rco- > 00.txt > > On Mon, Dec 8, 2014 at 10:22 PM, Pankaj Garg > <[email protected]> wrote: > > > > > >> -----Original Message----- > >> From: nvo3 [mailto:[email protected]] On Behalf Of Tom Herbert > >> Sent: Monday, December 8, 2014 9:50 PM > >> To: Darrel Lewis (darlewis) > >> Cc: [email protected] > >> Subject: Re: [nvo3] New Version Notification for > >> draft-herbert-vxlan-rco- 00.txt > >> > >> On Tue, Dec 2, 2014 at 12:34 PM, Darrel Lewis (darlewis) > >> <[email protected]> wrote: > >> > setting it to 0 will reduce it by 100% > >> > > >> I assume you mean setting the outer UDP checksum to zero. The purpose > >> of remote checksum offload is to offload an inner encapsulated > >> checksum. For example, to offload the TCP checksum in an > >> IP/UDP/VXLAN/Eth/IP/TCP encapsulation. Many commonly deployed > NICs do > >> not support checksum offload for such an encapsulation, which means > >> if a host is sourcing or receiving encapsulated TCP packets they will > >> need to compute TCP checksum in the CPU-- this is an expensive per byte > cost. > > > > [PG] Legacy NICs don't support large send offload or > VMQ/VMDq/NetQueue > > for encapsulated packets either and those are far more important from > > performance and/or CPU overhead reduction perspective than checksum > > offload. I am not sure if it is practical to even use NICs in network > > virtualization scenarios without support of these offloads, given how > > poor the performance is, at least in 10G environment > > Hi Pankaj, > > From my experience it is practical. We already have a large scaled > deployment that does not use any specialized NIC virtualization features. It > would be an enormous cost to upgrade all of the NICs to support > virtualization (e.g. host bypass like you mentioned), and even if we wanted > to do that I don't believe there are currently any available that would meet > our security and control requirements, nor do we have an IETF protocol > standard for nvo3 encapsulation so when that materializes I worry that we'd > need change again. If we can leverage the existing features in legacy NICs > and otherwise optimize the data path, performance actually can come pretty > close to that of native networking.
[PG] My point was that I am not sure if just reducing checksum overhead would be sufficient to make network virtualization performance acceptable to most people. In my experience with Hyper-V in both Azure and private clouds, it wasn't acceptable, since the performance penalty was just too high and we had to "opportunistically" use other techniques like NAT with legacy NICs to simulate "network virtualization" with acceptable performance. If legacy NICs are important, then I feel it would be better to think of an overlay solution that can work with such NICs and still provide reasonable performance e.g. something like STT, maybe? As I am not sure if the gains we get with remove checksum offload moves the needle much (without LSO and VMQ) but then, if you _have_ a reason to use VXLAN with legacy NICs, then yes, it would save some CPU cycles. > > I suspect for a long time there will be other people in the same boat in > running virtualization in their DC on legacy hardware. VXLAN is not > specifically > a hardware protocol, so I think it's reasonable to implement open techniques > to make it work well as possible in these environments. > > Tom > > > (which should be common in datacenters). So I am not sure how much > > value there is In optimizing for this. > > > >> > >> Remote checksum offload addresses this by enabling the outer UDP > >> checksum and inferring the value of the inner checksum based on that > >> on receive. The ability to do checksum offload for UDP/IP packets is > >> pretty ubiquitous in deployed NICs. With UDP checksum enabled and > >> remote checksum offload we generally see much better performance in > >> host to host encapsulation than setting UDP checksum to zero. > >> > >> As I also mentioned in the draft, enabling the UDP checksum covers > >> more of the packet including IP pseudo header and virtual network ID. > >> Covering the pseudo header is especially important in IPv6 since > >> there is no header checksum in IPv6 (there is ongoing discussion > >> about the importance of UDP checksum in tunneling with IPv6). > >> > >> Thanks, > >> Tom > >> > >> > -D > >> > On Dec 2, 2014, at 9:47 AM, Tom Herbert <[email protected]> > wrote: > >> > > >> >> This is a proposal for remote checksum offload in VXLAN. We use a > >> >> compressed format for the remote checksum offload data to support > >> >> offload of UDP and TCP inner checksums. This allocates a reserved > >> >> bit > >> >> (#10 which would be immediately after version bits in VXLAN-GPE) > >> >> and uses the 8 bits after VNI for data. > >> >> > >> >> I have implemented this in Linux stack and posted a first cut on > >> >> netdev (http://permalink.gmane.org/gmane.linux.network/339847). It > >> >> works as expected, eliminating the need to perform host checksum > >> >> computation over a packet when encapsulating TCP over VXLAN > >> >> between hosts using plain NICs. In simple throughput tests this > >> >> reduced CPU usage by ~20% on transmit, and using outer UDP > >> >> checksum reduces CPU on receive by almost 50%. > >> >> > >> >> Comments are appreciated. > >> >> > >> >> Thanks, > >> >> Tom > >> >> > >> >> > >> >> ---------- Forwarded message ---------- > >> >> From: <[email protected]> > >> >> Date: Tue, Dec 2, 2014 at 8:13 AM > >> >> Subject: New Version Notification for > >> >> draft-herbert-vxlan-rco-00.txt > >> >> To: Tom Herbert <[email protected]> > >> >> > >> >> > >> >> > >> >> A new version of I-D, draft-herbert-vxlan-rco-00.txt has been > >> >> successfully submitted by Tom Herbert and posted to the IETF > >> >> repository. > >> >> > >> >> Name: draft-herbert-vxlan-rco > >> >> Revision: 00 > >> >> Title: Remote checksum offload for VXLAN > >> >> Document date: 2014-12-01 > >> >> Group: Individual Submission > >> >> Pages: 6 > >> >> URL: > >> >> http://www.ietf.org/internet-drafts/draft-herbert-vxlan-rco-00.txt > >> >> Status: > >> >> https://datatracker.ietf.org/doc/draft-herbert-vxlan-rco/ > >> >> Htmlized: http://tools.ietf.org/html/draft-herbert-vxlan-rco-00 > >> >> > >> >> > >> >> Abstract: > >> >> This specification describes remote checksum offload for VXLAN. > >> >> Remote checksum offload is a mechanism that provides checksum > >> offload > >> >> of transport checksums in encapsulated packets using rudimentary > >> >> offload capabilities found in most Network Interface Card (NIC) > >> >> devices. The outer UDP checksum is enabled on transmit and, with > some > >> >> additional meta data, a receiver is able to deduce the checksum to be > >> >> set in an encapsulated packet. Effectively this offloads the > >> >> computation of the inner checksum which can be a significant > >> >> performance optimization. Enabling the UDP checksum has the > >> >> additional advantage that it covers more of the packet including the > >> >> IP pseudo header and virtual network identifier. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> Please note that it may take a couple of minutes from the time of > >> >> submission until the htmlized version and diff are available at > >> tools.ietf.org. > >> >> > >> >> The IETF Secretariat > >> >> > >> >> _______________________________________________ > >> >> nvo3 mailing list > >> >> [email protected] > >> >> https://www.ietf.org/mailman/listinfo/nvo3 > >> > > >> > >> _______________________________________________ > >> nvo3 mailing list > >> [email protected] > >> https://www.ietf.org/mailman/listinfo/nvo3 _______________________________________________ nvo3 mailing list [email protected] https://www.ietf.org/mailman/listinfo/nvo3
