Hi Tobias, We already talked here in Bangkok and I’m willing to discuss more on IPv6 fragments for AS6830, AS33915, AS6730, AS6848 or AS5089. I’ll be in the dnsop meeting in person in Bangkok. I’m looking for any BCP guidance to mitigate DDOS attack vectors by IPv6 fragmentation while not filtering fragments that will break DNS. I think 3901bis will probably be a good place to give that guidance, but at the same time we need to be looking for any DDoS mitigation guidance outside DNSOP that might be currently conflicting with this DNS only guidance. Any references to current guidance on IPv6 fragments is welcome so we will know what competing guidance we will also need to adjust or reference in 3901bis.
Antoin Verschuren Senior Manger Network Security Liberty Global > Op 7 mrt. 2025, om 10:53 heeft Tobias Fiebig > <tobias=40fiebig...@dmarc.ietf.org> het volgende geschreven: > > Hello Geoff, > > On Thu, 2025-03-06 at 01:24 +0000, Geoff Huston wrote: >> There is a difference between taking a single vantage point (or a >> small set of vantage) and looking at the behaviour of a resolution of >> a large domain set, >> and taking a very large set of vantage points and looking at the >> behaviours of resolution of a small set of domains that exhibit >> particular properties. >> >> The work we do at APNIC Labs is focussed on the latter mode, and this >> has lead to our observations about the incremental issues arising >> from the use of IPv6 transport for the DNS. > > Actually, you are not using more vantage points; Given that you are > inverting the measurement direction, in your setup, the authoritatives > are the vantage points. > >> Clearly we disagree here over the quantification of this >> impactarising from these differing methodologies. > > The problem is, that your work is very much impossible to reproduce, > and raw data is not available. That makes it impossible to argue on > this data, because _any_ other measurement could ultimately be argued > against with 'but our data does not show this'. > > In fact, your measurements demonstrate several limitations highly > impacting the results and conclusions. Those events range in the same > order of magnitude as the _whole effects_ I am measuring. In fact, for > many cases, it seems like core effects are tied to an individual > operator; Btw, does somebody know someone working for LGI/Liberty > Global? Fixing their fragment dropping would make a lot of things a lot > better in those stats. > > --- > # Measured fragment drop rates > > Given what we are discussing, fragmentation dropping as a major > concern, this ist the most critical issue in the data. During my scans > of authoritatives, the impact of fragments being lost was not notable > for .nl. > > .nl saw between 0.25% and 2.5% of connections during resolution > encounter fragments on UDP, with a lower share of fragments for IPv6 > enabled NSSets. Naturally, fragmentation was higher for zones with > DNSSEC, but not notably different between v4 and v6: > > https://data.measurement.network/dns-mtu-msmt/out_data/2025-03-04/plots/pcap_stats/pcap_stats_UDP_frag.pdf > > At the same time, the APNIC data reports ~40% drop rate for IPv6 > fragments in the Netherlands. Notably, up until the second half of > 2023, that rate was well below 5%. After an event at the end of August > 2023, where both frag drop and atomic frag drop rate rose to ~40%, the > rate seemed to plateau around 15%, followed by a steep increase in the > end of November 2023, again to a rate around 40%, comparable to the > high increase observed in the end of August 2023. > > Unlike two other constraint events since then, this is not accompanied > by a corresponding decrease in the number of clients. > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/frag-drop-nl.png > > Listing this by AS, it is clear that LGI/AS33915 is the main culprit > here, providing Internet for a large portion of people in the > Netherlands via its subsidiary Ziggo. However, this aggregation > approach highly skews the country stats; We have to convince one > operator to properly handle fragments, not half a country. > > Furthermore, this pattern continues across countries: > > For PL, IE, SK, the root cause is AS6830 (LGI). > For CH, the root cause is AS6730 (Sunrise, behind LGI). > For BE, the root cause is AS6848 (Telenet BV, behind LGI). > ... > > > Interestingly, though, the LGI/Vodafone instance in DE (AS3209) seems > to _not_ have such a high frag drop rate. At the same time, Cloudflare- > -likely due to their tunnel offerings--seems to have a notable impact > in the eyeball space. > > Hence, I would argue that we do not really have a global problem with > IPv6 fragment dropping (at least in Europe). We have an issue with > LGI's network (or a specific device/vendor/configuration used only in > specific countries). Does somebody know someone working there? > > > # Impact of vantage point location in a fragment-dropping AS > > Interestingly, though, AS63949 AKAMAI-LINODE-AP Akamai Connected > Cloud, shows up prominently in the data for .NL; According to the APNIC > data, this AS shows a volatile drop rate, averaging well above 50%. > > This is also the AS in which the vantage points seem to be located. It > is unclear, in how far this additionally impacts the measurements' > reliability. > > --- > > Beyond those aspects, there are several other aspects which raise some > doubts w.r.t. the reliability of the meausrement platform: > > # Noise Effects > > The measurements exhibit strange cliffs with volatile jumps in the > range of orders of magnitude impacting all measured endpoints with no > clear root-cause; See, for example, the suspicious cliff in q1 use from > late 2022 until late 2024: > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/dns-resolver-use-all-resolvers.png > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/dns-resolver-use-first.png > > Similar noise effects occur in other measurements as well, for example > the RPKI validation data (for example for AS3320's announced prefixes): > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/AS3320-RPKI.png > > > # High participation from North Korea > > Some time ago I remarked that I found the high number of daily users > from NK a bit odd; Indeed, it seems like KP is no longer listed as part > of Eastern Asia: > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/adds_eastern_asia.png > > However, looking at the country-site for 'KP', that data still seems to > be there, also receiving a comparatively high 'Weight' for the number > of samples: > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/KP-samples-over-time.png > > Interestingly, the AS responsible for this has actually less v4 > announced than me, and seems to also contain a fair bit of hosted > services in those prefixes: > > https://bgp.tools/prefix/175.45.179.0/24#dns > > https://bgp.tools/prefix/175.45.176.0/24#dns > > For comparison, look at, e.g., FFMUC: > > https://rincewind.home.aperture-labs.org/~tfiebig/mtu/FFMUC-samples-over-time.png > > This eyeball project as ~2-4k concurrently active users; See: > > https://stats.ffmuc.net/d/kUoZ2DRWz/network-overview?orgId=1&from=now-7d&to=now&timezone=browser&refresh=1m > > > # Impact of MTU mingeling from hypervisors > > As I noted before, the set of vantage points for your measurements is > actually somewhat limited, effectively to the number (iirc <=10) > authoritatives put up. Furthermore, as far as I know, you are running > the authoritatives on virtual machines rented from a commodity hoster. > > The problem there is that the use of VirtIO NICs can actually heavily > impact MTU behavior in a way transparent to the virtual machine. The > only way I managed to force my systems to fragment and drop 'correctly' > was not only forcing TSO/GRO et al. to off on the host, but also > explicitly doing so on the nic of the Hypervisor. > > # Zone-path of measurement domain > > One of the domains involved in the measurements seems to be > dotnxdomain.net: > > % dig +short NS dotnxdomain.net > ns1.dotnxdomain.net. > ns3.dotnxdomain.net. > > > At least from an EU PoP, these do resolve to IPs in Australia (ns1.) > and SoCal (ns3.); > > % dig +short A ns1.dotnxdomain.net > 203.133.248.6 > % dig +short A ns3.dotnxdomain.net > 173.230.146.214 > % dig +short AAAA ns1.dotnxdomain.net > 2401:2000:6660::6 > % dig +short AAAA ns3.dotnxdomain.net > 2600:3c01::f03c:93ff:fe02:be25 > > This induces notable latency and sources of error along the path, even > if the final resolution for the <cc/region> names thereunder used in > the actual measurements are different. Essentially, it adds 'traversing > half the planet' to the set of noise sources. > > # Measurement Setup / Config Location > > The measurements seem to (have been) configured via > cfg.dotnxdomain.net; This name (again, via the aforementioned NS) > resolves to three distinct IPs, located in AP, EU, and NA: > > % dig +short A cfg.dotnxdomain.net > 139.162.2.194 > 139.162.149.100 > 45.79.7.112 > > % dig +short AAAA cfg.dotnxdomain.net > 2a01:7e01::f03c:91ff:fe12:6bfe > 2400:8901::f03c:91ff:fe98:63d6 > 2600:3c00::f03c:91ff:fe98:16c8 > > The problem with that is that clients may end up on a rather remote > cfg. host of these. That, in turn, may lead to self-selection depending > on clock and timeout settings. At least for cfg. retrieval, the timeout > seems to be 10000ms, which should be mostly save. Nevertheless, the > added latency may lead clients to abandon the ad; Also, I am not sure > what timeout behavior modern browsers are forcing. > > When throwing these names into Atlas with resolve-on-probe, at least, > RTT and hit end-point vary wildly and are not congruent with the > physical locations of probes: > > v4: > https://atlas.ripe.net/measurements/89419109/overview > > v6: > https://atlas.ripe.net/measurements/89419368/overview > > --- > > So, in summary, I am not convinced that the data commonly referenced is > robust enough to allow the strength of conclusions drawn on fragment > handling for IPv6. > > With best regards, > Tobias > > > _______________________________________________ > DNSOP mailing list -- dnsop@ietf.org > To unsubscribe send an email to dnsop-le...@ietf.org _______________________________________________ DNSOP mailing list -- dnsop@ietf.org To unsubscribe send an email to dnsop-le...@ietf.org
