On Mon, Dec 18, 2023 at 8:15 AM Tim Chown <Tim.Chown=40jisc.ac...@dmarc.ietf.org> wrote: > > Hi, > > Apologies for the delay in posting these notes. Gorry and I held a side > meeting in Prague on the topic of (lack of) use of jumbo frames, and what > topics might lie within the IETF’s remit to help promote greater use. > > After talking to an AD it was suggested we raise the topic on the int-area > list to gauge interest, rather than set up a new list at this stage. > > So, all thoughts and comments welcome...
Sorry I missed the meeting, I have a couple of comments below... > > -- > > Jumbo frame side meeting, IETF118, 2-3pm Thu 9 November > > Convened by Tim Chown (Jisc) and Gorry Fairhurst (Univ Aberdeen) > > The meeting had no set agenda. The aim was to gather those interested in more > widespread use of jumbo frames to gather and discuss what actions might be > taken in or by the IETF and its WGs towards that goal. > > Comments: > • There is no standard for Ethernet for frame sizes above 1500 bytes > • Would it be useful to work towards a “certified jumbo” interoperability > test? > • NICs at 1Gbit/s+ should all use phase-locked loop (PLL). > • What tools should we use to identify issues or errors in transmission > at various MTU sizes? > • Tim noted that Jisc’s 100G perfSONAR node at London showed no errors on > its 9000 MTU interface – stats can be seen under the interface details > section at https://ps-london-bw.perf.ja.net/toolkit/ > • We should consider the relevance of MTU in respective IETF areas – INT, > TSV and OPS > • Jen Linkova has talked about networks with multiple sizes of MTU > • There are providers who offer 9000 MTU networks, end-to-end, such as > Hurrican Electric > • Tim reported that many PBs of data are moved by the CERN experiments > and a proportion of that is using 9000 MTU. Single stream TCP performance > can be 2-3x better, depending on RTT and other factors. This is going to be very dependent on implementation as well, particularly how effective GSO and GRO are and whether they are offloaded to hardware in TSO and LRO. > • What issues might there be in specific technologies, e.g. ND, BGP, > ECMP, multipath TCP, …? > • There is a perception that IXPs find 9000 MTU problematic > • There are previous IETF I-Ds on MTU use, e.g. in IXPs – we should look > at old drafts or any RFCs > • There may be relevant presentations from *NOG and RIR member meetings > • Improvements to host stacks can make the performance gains of jumbo > frames less important, e.g. various offloading technologiesCan we get current > measurements and data, e.g., via MAPRG? To a degree techniques like GSO and GRO help. But these are dependent on specific transport protocols. > • We should look at hyperscalers; there is support there for 9000 MTU I believe at least Google is using 9K MTUs in the datacenter for a while. This allows two 4K pages to be sent in TCP which maps to the OS concept of pages. When combined with header data split, this allows zero copy receive to user space via page flipping thereby eliminating expensive copies from the kernel to user space. Interestingly, there is a deployed use case of IPv6 jumbograms; this described in https://lwn.net/Articles/884104/ https://netdevconf.info/0x15/slides/35/BIG%20TCP.pdf. The basic idea is that the host stack can create TCP segments larger than 64K. GSO or TSO is always used to break up the jumbo segments into MTU sized segments for sending. In this way the jumbogram is never sent on the wire, but we get the performance advantages since it saves processing in the TCP stack. > • IPsec, and any encapsulation that benefits from avoiding fragmentation, > can work better with jumbo frames > • We could look a Globus transfer logs to detect MD5 errors for evidence > of issues in the application data not picked up at lower layers > • There are other non-Ethernet technologies used in DCs with large frames > • Does QUIC break offload due to its encryption? In practice QUIC uses a > Max Datagram Packet size less than 1500. Might larger MTUs be useful for QUIC > • Post-quantum scenarios were mentioned. > • What about MTU discovery? There is anecdotal evidence of issues; Tim > has seen this at a UK university where ICMPv6 PTB was being dropped. It's generally assumed that ICMP, including PTB, is unreliable. > • PLPMTUD is specified by QUIC; useful when there’s no path back to a > sender for receipt of an ICMP PTB message. > > Agreed actions: > • Tim will ask Eric Vyncke (INT area AD) for support to create a > “jumbo-discuss” IETF mail list > • We will seek to collectively document the status of jumbo frames, > focusing on what works (success stories), opportunities, gaps (potential work > items in the IETF and elsewhere) and other open issues. > • Tim will ask Eric Vyncke for a side meeting at a future IETF. > • We will seek to present relevant parts of the above documented status > in the INT, TSV and OPS area open meetings at the next IETF meeting. > • Tim will email the 118attendees list with the meeting notes > > — > > Tim > _______________________________________________ > Int-area mailing list > Int-area@ietf.org > https://www.ietf.org/mailman/listinfo/int-area _______________________________________________ Int-area mailing list Int-area@ietf.org https://www.ietf.org/mailman/listinfo/int-area
