Re: QUIC traffic throttled on AT&T residential

[Dan Wing]

Yes, other than AT&T increasing their permitted incoming UDP traffic -- the 
easiest thing AT&T can do -- AT&T could ask the vendor of their flow 
restricting device to use bi-directional UDP traffic on same 5-tuple to 
indicate "desire to receive", rather than solely examining incoming UDP traffic 
as they are doing today; this is not easy but also not impossible.

While it is true Youtube/Google could treat AT&T-sourced connections as 'bad' 
and force TCP, but I am sure Youtube/Google is already gathering those 
statistics and already aware that AT&T is throttling.  For all we know, you and 
the others noticing the issue have fallen into the pit of A/B testers checking 
for their current throttling, and others aren't being throttled.  Perhaps it's 
everyone, the tests are not well described or well announced.  Youtube/Google 
is hoping customers complain to AT&T so that AT&T removes or improves the flow 
restrictor, because otherwise AT&T customers won't be able to get QUIC.  
Similarly, AT&T could protect their users from AT&T's own rate limiting by 
blocking QUIC towards major servers that support QUIC but such blocking becomes 
problematic as QUIC rolls out beyond Google to Cloudflare and elsewhere.

This tussle has similarities to IPv6 vs IPv4.

-d


> On Feb 18, 2020, at 4:00 PM, Ca By  wrote:
> 
> 
> 
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling  > wrote:
> I've AT&T fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
> 
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
> 
> I assume traffic is being throttled on AT&T's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
> 
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
> 
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
> 
> May I naively ask if Google staff are working with AT&T to address this?
> 
> Sounds like you found the answer, ATT just needs to scale up your rejection 
> approach that is proven to work well. 
> 
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they 
> would be mixing with bad company in the pool of ipv4 udp traffic ... but they 
> have reasons. 
> 
> I am not a fan of quic or any udp traffic. My suggestion was that Google use 
> an new L4 instead of UDP, but that was too hard for the Googlers. 
> 
> So here we are.
> 
> Just say no to udp. 
> 
> 
> 
> 
> -- Dan

Re: QUIC traffic throttled on AT&T residential

[Dan Wing]

There are choices, such as making connection initiation, connection acceptance, 
and connection termination parsable by network elements on the path so state 
can be established, maintained, and cleared, DoS can be identified, and so on.  
The decision was to hide all that from network elements.

-d


> On Feb 20, 2020, at 7:54 PM, Matthew Kaufman  wrote:
> 
> 
> 
> On Thu, Feb 20, 2020 at 8:10 AM Ca By  > wrote:
> 
> 
> Not indiscriminate. 
> 
> As Google was informed by network operators all along since 2014, ipv4 UDP is 
> a major uptime threat via DDoS to access networks.  
> ...
> 
> Google choose not to be sensitive to that, they were told where the landmines 
> were, they choose to go on anyhow. 
> 
> 
> It isn’t like they had a choice. You can’t build a transport protocol like 
> QUIC on top of TCP (I know, I built one of its ancestors, which also uses UDP 
> underneath). And if you think getting UDP through existing networks is hard, 
> try using a novel IP protocol number.
> 
> Matthew Kaufman
> 

RE: Problems with removing NAT from a network

[Dan Wing]

> -Original Message-
> From: Matthew Kaufman [mailto:matt...@matthew.at]
> Sent: Thursday, January 06, 2011 6:55 PM
> To: Owen DeLong
> Cc: Nanog Operators' Group
> Subject: Re: Problems with removing NAT from a network
> 
> On 1/6/2011 5:48 PM, Owen DeLong wrote:
> > Doesn't all of this become moot if Skype just develops a dual-stack
> capable client
> > and servers?
> >
> Not really. Imagine the case where you're on IPv6 and you can only
> reach
> IPv4 via a NAT64, and there's no progress made on the detection
> problem.
> And your family member is on a Skype-enabled TV plugged into an
> IPv4-only ISP.
> 
> Now you can't get a direct media path between you, even though their
> ISP
> is giving them IPv4 and your ISP is *claiming* you can "still reach the
> IPv4 Internet".
> 
> Skype can still make this work by relaying,

Skype could make it work with direct UDP packets in about 92% of
cases, per Google's published direct-to-direct statistic at
http://code.google.com/apis/talk/libjingle/important_concepts.html

-d


> but in order to protect the
> relay machine's bandwidth it will rate-limit the traffic, and so your
> A/V experience will suffer. And that's assuming there's enough
> dual-stacked relays... if there aren't, it won't be possible to find a
> relay that they can reach over IPv4 and you can reach over IPv6 that
> has
> available bandwidth.
> 
> Matthew Kaufman

RE: Problems with removing NAT from a network

[Dan Wing]

> On 1/6/2011 9:28 PM, Dan Wing wrote:
> >> -Original Message-
> >> From: Matthew Kaufman [mailto:matt...@matthew.at]
> >> Not really. Imagine the case where you're on IPv6 and you can only
> >> reach
> >> IPv4 via a NAT64, and there's no progress made on the detection
> >> problem.
> >> And your family member is on a Skype-enabled TV plugged into an
> >> IPv4-only ISP.
> >>
> >> Now you can't get a direct media path between you, even though their
> >> ISP
> >> is giving them IPv4 and your ISP is *claiming* you can "still reach
> the
> >> IPv4 Internet".
> >>
> >> Skype can still make this work by relaying,
> > Skype could make it work with direct UDP packets in about 92% of
> > cases, per Google's published direct-to-direct statistic at
> > http://code.google.com/apis/talk/libjingle/important_concepts.html
> >
> If one end is behind a NAT64 and there is no mechanism for discovering
> the NAT64's IPv6 interface prefix and mapping algorithm (and at present
> there is not), there is no way to send IPv6 IP packets from the
> IPv6-only host to IPv4 literal addresses (that is to say, addresses
> learned via a mechanism other than DNS responses synthesized by the
> DNS64 part of the NAT64 "solution") on the IPv4 Internet through said
> NAT64.
> 
> That's the case we're discussing here.

There are a bunch of ideas for how to accomplish that.  Several of
the ideas don't require any support by the network infrastructure,
draft-korhonen-behave-nat64-learn-analysis.

> It breaks Skype, Adobe's RTMFP, BitTorrent, ICE-based NAT traversal,
> etc. Even the protocol described in the referenced document, Jingle (as
> it essentially uses ICE) fails. The candidate IPv4 addresses for the
> end
> that's on the IPv4 Internet (local and STUN-derived) that are delivered
> over Jingle's XMPP path cannot be used by the host that is on IPv6 +
> NAT64 to reach the IPv4 Internet because it has no IPv4 sockets
> available to it and even if it knew that NAT64 existed (which would
> take
> a modification to the Jingle-based apps) and opened an IPv6 socket it
> wouldn't know what IPv6 address to use to reach the IPv4 host because
> there's no discovery mechanism. If you want we can take this back to
> the BEHAVE list now.

Sure.

-d


> 
> Matthew Kaufman

Re: real-world data about fragmentation

[Dan Wing]

On Apr 2, 2014, at 11:14 AM, Joe Abley  wrote:

> Hi all,
> 
> It's common wisdom that a datagram that needs to be fragmented between 
> endpoints (because it is bigger than the path MTU) will demonstrate less 
> reliable delivery and reassembly than a datagram that doesn't need to be 
> fragmented, because math, firewall, other, take your pick.
> 
> Is anybody aware of any wide-scale studies that examine the probability of 
> fragmentation of datagrams of different sizes?
> 
> For example, I could reasonable expect an IPv4 packet of 576 bytes not to be 
> fragmented very often (to choose a size not at random). The probability of a 
> 10,000 octet IPv4 packet getting fragmented seems likely to be 100%, if we're 
> talking about arbitrary paths across the Internet.
> 
> What does the curve look like between 576 bytes and 10,000 bytes?
> 
> I might expect exciting curve action around 1500 bytes (because ethernet), 
> 1492 (PPPoE), 1480 (GRE), etc. But I'm interested in actual data.
> 
> Anybody have any pointers? IPv4 and IPv6 are both interesting.

Seems a good thing for RIPE Atlas probes to measure.  But they are probably not 
generally connected to representative networks (read: poor networks).

-d

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Geoff Huston [mailto:g...@apnic.net]
> Sent: Wednesday, September 07, 2011 10:27 PM
> To: Leigh Porter
> Cc: nanog@nanog.org list; Daniel Roesen
> Subject: Re: NAT444 or ?
> 
> 
> On 08/09/2011, at 2:41 AM, Leigh Porter wrote:
> 
> >
> >
> >> -Original Message-
> >> From: Daniel Roesen [mailto:d...@cluenet.de]
> >> Sent: 07 September 2011 17:38
> >> To: nanog@nanog.org
> >> Subject: Re: NAT444 or ?
> >>
> >> On Wed, Sep 07, 2011 at 12:16:28PM +0200, Randy Bush wrote:
>  I'm going to have to deploy NAT444 with dual-stack real soon now.
> >>>
> >>> you may want to review the presentations from last week's apnic
> >> meeting
> >>> in busan.  real mesurements.  sufficiently scary that people who
> were
> >>> heavily pushing nat444 for the last two years suddenly started to
> say
> >>> "it was not me who pushed nat444, it was him!"  as if none of us
> had
> >> a
> >>> memory.
> >>
> >> Hm, I fail to find relevant slides discussing that. Could you please
> >> point us to those?
> >>
> >> I'm looking at http://meetings.apnic.net/32
> >
> > There is a lot in the IPv6 plenary sessions:
> >
> > http://meetings.apnic.net/32/program/ipv6
> >
> > This is what I am looking at right now. Randy makes some good
> comments in those sessions. I have not found anything yet, but I am
> only on session 3, pertaining specifically to issues around NAT444.
> 
> It may not be what Randy was referring to above, but as part of that
> program at APNIC32 I reported on the failure rate I am measuring for
> Teredo. I'm not sure its all in the slides I was using, but what I was
> trying to say was that STUN is simply terrible at reliably negotiating
> a NAT.

Teredo does not use STUN; Teredo was implemented before STUN was fully
spec'd and does its own thing.

Teredo tries to determine if the type of NAT it is behind ("cone", 
"symmetric", etc.)  Determining the type of NAT has been found to 
be difficult, and nearly impossible (*) and removed from the current
RFC for STUN (**).  I suspect most of Teredo's failures are related 
to this procedure not working effectively.  A CGN can't improve that.

(*) http://tools.ietf.org/html/rfc5780#section-1
(**) http://tools.ietf.org/html/rfc5389#section-2

> I was then wondering what pixie dust CGNs were going to use that
> would have any impact on the ~50% connection failure rate I'm observing
> in Teredo.  And if there is no such thing as pixie dust (damn!) I was
> then wondering if NATs are effectively unuseable if you want anything
> fancier than 1:1 TCP connections (like multi-user games, for example).
> After all, a 50% connection failure rate for STUN is hardly encouraging
> news for a CGN deployer if your basic objective is not to annoy your
> customers.

If the CGN has both Endpoint-Independent Filtering (***) behavior
and Endpoint-Independent Mapping () behavior, the CGN won't make
Teredo worse -- Teredo will be as bad as today, caused by the home
user's own pretty NAT.  With that behavior, it also won't make 
applications that use STUN or ICE worse, or applications that use 
STUN-like or ICE-like techniques such as Skype.

(***) endpoint-independent filtering: means it doesn't filter incoming
packets after a mapping is created.  See
http://tools.ietf.org/html/rfc4787#section-5 for canonical definition.

() Endpoint-Independent Mapping:  means when the internal host sends a
packet with the same source port, to any destination, the same public port
is mapped.   See http://tools.ietf.org/html/rfc4787#section-4.1 for
canonical definition

-d

RE: NAT444 or ?

[Dan Wing]

...
> The striking thing I picked up is that NTT considers the CGN equipment
> a big black hole where money goes into. Because it won't solve their
> problem now or in the future and it becomes effectively a piece of
> equipment they need to buy and then scrap "soon" after.

It would get scrapped when all servers support dual stack.  What year
is that predicted to occur?

> They acknowledge the need, but they'd rather not buy one.
> That and they (the isp) get called for anything which doesn't work.

-d

RE: what about the users re: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Christian de Larrinaga [mailto:c...@firsthand.net]
> Sent: Thursday, September 08, 2011 8:05 AM
> To: Cameron Byrne
> Cc: NANOG
> Subject: what about the users re: NAT444 or ?
> 
> I wonder if the discussion as useful as it is isn't forgetting that the
> edge of Internet has a stake in getting this right too! This is not
> just an ISP problem but one where content providers and services that
> is the users need to get from here to there in good order.
> 
> So
> 
> What can users do to encourage ISPs to deploy v6 to them?
> What can users do to ease the pain in reaching IPv4 only sites once
> they are on IPv6 tails?
> 
> Is there not a bit of CPE needed here? What should the CPE do? and not
> do? should it deprecate NAT/PAT when it receives 1918 allocation from a
> CGN?

Careful with that idea -- people like their in-home network to continue
functioning even when their ISP is down or having an outage.  Consider
a home NAS holding delivering content to the stereo or the television.
It is possible to eliminate reliance on the ISP's network and still
have the in-home network function, but it's more difficult than just
continuing to run NAT44 in the home like today.  (Dual Stack-Lite
can accomplish this pretty easily, because the IPv4 addresses in
the home can be any IPv4 address whatsoever -- which allows the
in-home CPE ("B4", in Dual Stack-Lite parlance) to assign any address
it wants with its built-in DHCP server.)

-d

> and less technically but relevant I think is to ask about cost? who
> pays?
> 
> 
> Christian
> 
> On 8 Sep 2011, at 15:02, Cameron Byrne wrote:
> 
> > On Sep 8, 2011 1:47 AM, "Leigh Porter" 
> wrote:
> >>
> >>
> >>
> >>> -Original Message-
> >>> From: Owen DeLong [mailto:o...@delong.com]
> >>> Sent: 08 September 2011 01:22
> >>> To: Leigh Porter
> >>> Cc: Seth Mos; NANOG
> >>> Subject: Re: NAT444 or ?
> >>>
>  Considering that offices, schools etc regularly have far more than
> 10
> >>> users per IP, I think this limit is a little low. I've happily had
> >>> around 300 per public IP address on a large WiFi network, granted
> these
> >>> are all different kinds of users, it is just something that
> operational
> >>> experience will have to demonstrate.
> 
> >>> Yes, but, you are counting individual users whereas at the NAT444
> >>> level, what's really being counted is end-customer sites not
> individual
> >>> users, so the term
> >>> "users" is a bit misleading in the context. A given end-customer
> site
> >>> may be from 1 to 50 or more individual users.
> >>
> >> Indeed, my users are using LTE dongles mostly so I expect they will
> be
> > single users. At the moment on the WiMAX network I see around 35
> sessions
> > from a WiMAX modem on average rising to about 50 at peak times. These
> are a
> > combination of individual users and "home modems".
> >>
> >> We had some older modems that had integrated NAT that was broken and
> > locked up the modem at 200 sessions. Then some old base station
> software
> > died at about 10K sessions. So we monitor these things now..
> >>
> >>
> >>>
>  I would love to avoid NAT444, I do not see a viable way around it
> at
> >>> the moment. Unless the Department of Work and Pensions release
> their /8
> >>> that is ;-)
> 
> >>>
> >>> The best mitigation really is to get IPv6 deployed as rapidly and
> >>> widely as possible. The more stuff can go native IPv6, the less
> depends
> >>> on fragile NAT444.
> >>
> >> Absolutely. Even things like google maps, if that can be dumped on
> v6,
> > it'll save a load of sessions from people. The sooner services such
> as
> > Microsoft Update turn on v6 the better as well. I would also like the
> CDNs
> > to be able to deliver content in v6 (even if the main page is v4)
> which
> > again will reduce the traffic that has to traverse any NAT.
> >>
> >> Soon, I think content providers (and providers of other services on
> the
> > 'net) will roll v6 because of the performance increase as v6 will not
> have
> > to traverse all this NAT and be subject to session limits, timeouts
> and
> > such.
> >>
> >
> > What do you mean by performance increase? If performance equals
> latency, v4
> > will win for a long while still. Cgn does not add measurable latency.
> >
> > Cb
> >> --
> >> Leigh
> >>
> >>
> >>
> __
> >> This email has been scanned by the MessageLabs Email Security
> System.
> >> For more information please visit http://www.messagelabs.com/email
> >>
> __
> >>

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Leigh Porter [mailto:leigh.por...@ukbroadband.com]
> Sent: Wednesday, September 07, 2011 1:38 PM
> To: David Israel; nanog@nanog.org
> Subject: RE: NAT444 or ?
> 
> 
> 
> > -Original Message-
> > From: David Israel [mailto:da...@otd.com]
> > Sent: 07 September 2011 21:23
> > To: nanog@nanog.org
> > Subject: Re: NAT444 or ?
> >
> > On 9/7/2011 3:24 PM, Seth Mos wrote:
> > > I think you have the numbers off, he started with 1000 users
> sharing
> > the same IP, since you can only do 62k sessions or so and with a
> > "normal" timeout on those sessions you ran into issues quickly.
> > >
> >
> > Remember that a TCP session is defined not just by the port, but by
> the
> > combination of source address:source port:destination
> > address:destination port.  So that's 62k sessions *per destination*
> per
> > ip address.   In theory, this particular performance problem should
> > only
> > arise when the NAT gear insists on a unique port per session (which
> is
> > common, but unnecessary) or when a particular destination is
> > inordinately popular; the latter problem could be addressed by
> > increasing the number of addresses that facebook.com and google.com
> > resolve to.
> 
> Good point, but aside from these scaling issues which I expect can be
> resolved to a point, the more serious issue, I think, is applications
> that just do not work with double NAT. Now, I have not conducted any
> serious research into this, but it seems that draft-donley-nat444-
> impacts does appear to have highlight issues that may have been down to
> implementation.

Draft-donley-nat444-impacts conflates bandwidth constraints with CGN
with in-home NAT.  Until those are separated and then analyzed carefully,
it is harmful to draw conclusions such as "NAT444 bad; NAT44 good".

> Other simple tricks such as ensuring that your own internal services
> such as DNS are available without traversing NAT also help.

Yep.  But some users want to use other DNS servers for performance
(e.g., Google's or OpenDNS servers, especially considering they
could point the user at a 'better' (closer) CDN based on Client
IP), to avoid ISP DNS hijacking, or for content control (e.g.,
"parental control" of DNS hostnames).  That traffic will, necessarily,
traverse the CGN.  To avoid users burning through their UDP port 
allocation for those DNS queries it is useful for the CGN to 
have short timeouts for port 53.

> Certainly some more work can be done in this area, but I fear that the
> only way a real idea as to how much NAT444 really doe break things will
> be operational experience.

Yep.  (Same as everything else.)

-d


> 
> 
> --
> Leigh
> 
> 
> 
> 
> __
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> __

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Simon Perreault [mailto:simon.perrea...@viagenie.ca]
> Sent: Wednesday, September 07, 2011 2:29 PM
> To: nanog@nanog.org
> Subject: Re: NAT444 or ?
> 
> David Israel wrote, on 09/07/2011 04:21 PM:
> > In theory, this
> > particular performance problem should only arise when the NAT gear
> insists on a
> > unique port per session (which is common, but unnecessary)
> 
> What you're describing is known as "endpoint-independent mapping"
> behaviour. It
> is good for not breaking applications, not so good for scalability. RFC
> 4787 section 4.1 makes it a MUST.

There are two dimensions of that scalability, of course:

Endpoint-independent mapping means better scaling of the NAT itself, 
because it stores less state (slightly less memory for each active 
mapping and slightly less per-packet processing).  This savings
is exchanged for worse IPv4 utilization -- which I agree is not so
good for scalability.

-d

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: jean-francois.tremblay...@videotron.com [mailto:Jean-
> francois.tremblay...@videotron.com]
> Sent: Wednesday, September 07, 2011 10:06 AM
> To: d...@cluenet.de
> Cc: nanog@nanog.org
> Subject: Re: NAT444 or ?
> 
> On Wed, Sep 07, 2011 at 12:16:28PM +0200, Randy Bush wrote:
> > > I'm going to have to deploy NAT444 with dual-stack real soon now.
> > you may want to review the presentations from last week's apnic
> meeting
> > in busan.  real mesurements.  sufficiently scary that people who were
> > heavily pushing nat444 for the last two years suddenly started to say
> > "it was not me who pushed nat444, it was him!"  as if none of us had
> a
> > memory.
> >
> > Hm, I fail to find relevant slides discussing that. Could you please
> > point us to those?
> 
> I had the same question. I found Miyakawa-san's presentation has some
> dramatic examples of CGN NAT444 effects using Google Maps:
> http://meetings.apnic.net/__data/assets/file/0011/38297/Miyakawa-APNIC-
> KEYNOTE-IPv6-2011-8.pptx.pdf
> 
> 
> However these are with a very high address-sharing ratio (several
> thousands users per address). Using a sparser density (<= 64 users per
> address) is likely to show much less dramatic user impacts.

Try it at home.  With aggressive usage of Microsoft's Terraserver,
mapquest, or google maps, I'm able to burn through 120 or so 
TCP connections.  Move the map around, zoom in/out, enable/disable 
traffic, switch between satellite and map and overlay, repeat those
steps 2-3 times.  Don't be slow and don't wait for everything 
to paint.

Or crash your browser and when it restarts watch how many connections
it makes to re-open all your tabs.

I understand iTunes opens lots of connections, but I haven't looked
at that.

To experiment with limited ports at home, load 3rd party firmware 
onto your NAT -- most of them allow controlling the number 
of mappings (and by default, have higher limits than stock
firmware).  Or consume a bunch of your mappings with a 
script (such as the brain-dead Perl script below) and then 
start your testing.  Some NATs and some servers will kill the 
TCP sessions after inactivity (which is why I describe the 
script as brain-dead).

-d



#!/usr/bin/perl -w
use IO::Socket;

$max = shift(@ARGV);
my $count = 0;
my $host = shift(@ARGV) || "www.example.com";
my @remote;

print "connecting to $host\n";

while ($count < $max) {
printf ("connecting...(%d)\n", $count+1);
$remote[$count] = IO::Socket::INET->new(
Proto => "tcp",
PeerAddr => $host,
PeerPort => "80")
or warn "got an error";
$count++;
}

print "press Return to exit\n";
my $junk = ;

$count = 0;
while ($count < $max) {
close $remote[$count];
$count++;
}

exit;

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Randy Bush [mailto:ra...@psg.com]
> Sent: Wednesday, September 07, 2011 3:16 AM
> To: Leigh Porter
> Cc: North American Network Operators' Group
> Subject: Re: NAT444 or ?
> 
> > I'm going to have to deploy NAT444 with dual-stack real soon now.
> 
> you may want to review the presentations from last week's apnic meeting
> in busan.  real mesurements.  sufficiently scary that people who were
> heavily pushing nat444 for the last two years suddenly started to say
> "it was not me who pushed nat444, it was him!"  as if none of us had a
> memory.

Many of the problems are due to IPv4 address sharing, which will be
problems for A+P, CGN, HTTP proxies, and other address sharing 
technologies.  RFC6269 discusses most (or all) of those problems.
There are workarounds to those problems, but most are not 
pretty.  The solution is IPv6.

-d

RE: what about the users re: NAT444 or ?

[Dan Wing]

> One can do that with or without NAT. This claim that one cannot
> keep a network running without a service provider connected if you
> don't run NAT is a myth of dubious origin.

If the hosts are running DHCP, and the ISP is running the DHCP
server?  I guess they will fall back (after a while) to link-local
and continue on their merry way.

> > can accomplish this pretty easily, because the IPv4 addresses in
> > the home can be any IPv4 address whatsoever -- which allows the
> > in-home CPE ("B4", in Dual Stack-Lite parlance) to assign any address
> > it wants with its built-in DHCP server.)
> >
> 
> There are other ways to accomplish this as well.

-d

> > -d
> >
> >> and less technically but relevant I think is to ask about cost? who
> >> pays?
> 
> In some cases, ISPs will provide new CPE to their end users. In other
> cases,
> end-users will be expected to pay to upgrade their own.
> 
> Owen
> 
> >>
> >>
> >> Christian
> >>
> >> On 8 Sep 2011, at 15:02, Cameron Byrne wrote:
> >>
> >>> On Sep 8, 2011 1:47 AM, "Leigh Porter"
> 
> >> wrote:
> 
> 
> 
> > -Original Message-
> > From: Owen DeLong [mailto:o...@delong.com]
> > Sent: 08 September 2011 01:22
> > To: Leigh Porter
> > Cc: Seth Mos; NANOG
> > Subject: Re: NAT444 or ?
> >
> >> Considering that offices, schools etc regularly have far more
> than
> >> 10
> > users per IP, I think this limit is a little low. I've happily
> had
> > around 300 per public IP address on a large WiFi network, granted
> >> these
> > are all different kinds of users, it is just something that
> >> operational
> > experience will have to demonstrate.
> >>
> > Yes, but, you are counting individual users whereas at the NAT444
> > level, what's really being counted is end-customer sites not
> >> individual
> > users, so the term
> > "users" is a bit misleading in the context. A given end-customer
> >> site
> > may be from 1 to 50 or more individual users.
> 
>  Indeed, my users are using LTE dongles mostly so I expect they
> will
> >> be
> >>> single users. At the moment on the WiMAX network I see around 35
> >> sessions
> >>> from a WiMAX modem on average rising to about 50 at peak times.
> These
> >> are a
> >>> combination of individual users and "home modems".
> 
>  We had some older modems that had integrated NAT that was broken
> and
> >>> locked up the modem at 200 sessions. Then some old base station
> >> software
> >>> died at about 10K sessions. So we monitor these things now..
> 
> 
> >
> >> I would love to avoid NAT444, I do not see a viable way around
> it
> >> at
> > the moment. Unless the Department of Work and Pensions release
> >> their /8
> > that is ;-)
> >>
> >
> > The best mitigation really is to get IPv6 deployed as rapidly and
> > widely as possible. The more stuff can go native IPv6, the less
> >> depends
> > on fragile NAT444.
> 
>  Absolutely. Even things like google maps, if that can be dumped on
> >> v6,
> >>> it'll save a load of sessions from people. The sooner services such
> >> as
> >>> Microsoft Update turn on v6 the better as well. I would also like
> the
> >> CDNs
> >>> to be able to deliver content in v6 (even if the main page is v4)
> >> which
> >>> again will reduce the traffic that has to traverse any NAT.
> 
>  Soon, I think content providers (and providers of other services
> on
> >> the
> >>> 'net) will roll v6 because of the performance increase as v6 will
> not
> >> have
> >>> to traverse all this NAT and be subject to session limits, timeouts
> >> and
> >>> such.
> 
> >>>
> >>> What do you mean by performance increase? If performance equals
> >> latency, v4
> >>> will win for a long while still. Cgn does not add measurable
> latency.
> >>>
> >>> Cb
>  --
>  Leigh
> 
> 
> 
> >>
> __
>  This email has been scanned by the MessageLabs Email Security
> >> System.
>  For more information please visit http://www.messagelabs.com/email
> 
> >>
> __
> 
> >
> >

RE: NAT444 or ?

[Dan Wing]

> -Original Message-
> From: Owen DeLong [mailto:o...@delong.com]
> Sent: Tuesday, September 13, 2011 9:43 PM
> To: Dan Wing
> Cc: 'Leigh Porter'; 'David Israel'; nanog@nanog.org
> Subject: Re: NAT444 or ?
> 
> >>
> >> Good point, but aside from these scaling issues which I expect can
> be
> >> resolved to a point, the more serious issue, I think, is
> applications
> >> that just do not work with double NAT. Now, I have not conducted any
> >> serious research into this, but it seems that draft-donley-nat444-
> >> impacts does appear to have highlight issues that may have been down
> to
> >> implementation.
> >
> > Draft-donley-nat444-impacts conflates bandwidth constraints with CGN
> > with in-home NAT.  Until those are separated and then analyzed
> carefully,
> > it is harmful to draw conclusions such as "NAT444 bad; NAT44 good".
> >
> 
> Continuing to make this claim does not make it any more true.
> 
> Draft-donley took networks and measured their real-world functionality
> without NAT444, then, added NAT444 and repeated the same tests.
> Regardless of the underlying issue(s), the addition of NAT444 to the
> mix resulted in the forms of service degradation enumerated in the
> draft.

I disagree it reached that conclusion.  That may have been its
intent.

> Further, I would not ever say "NAT444 bad; NAT44 good". I would say,
> rather, "NAT44 bad, NAT444 worse". I think that's a pretty safe and
> non-harmful thing to say.

Yes, your statement is completely accurate.  I agree that IPv4 address 
sharing causes additional problems (which encompasses all forms of 
IPv4 address sharing), and CGN causes additional problems.

> >> Other simple tricks such as ensuring that your own internal services
> >> such as DNS are available without traversing NAT also help.
> >
> > Yep.  But some users want to use other DNS servers for performance
> > (e.g., Google's or OpenDNS servers, especially considering they
> > could point the user at a 'better' (closer) CDN based on Client
> > IP), to avoid ISP DNS hijacking, or for content control (e.g.,
> > "parental control" of DNS hostnames).  That traffic will,
> necessarily,
> > traverse the CGN.  To avoid users burning through their UDP port
> > allocation for those DNS queries it is useful for the CGN to
> > have short timeouts for port 53.
> >
> If the user chooses to use a DNS server on the other side of a NAT,
> then,
> they are choosing to inflict whatever damage upon themselves. I'm not
> saying that short UDP/53 timeouts are a bad idea, but, I am saying that
> the more stuff you funnel through an LSN at the carrier, the more stuff
> you will see break. This would lead me to want to avoid funneling
> anything
> through said NAT which I could avoid. Then again, I run my own
> authoritative and recursive nameservers in my home and don't use
> any NAT at all, so, perhaps my perspective is different from others.

Yeah, you are probably of about 1000 or maybe 3000 people in the 
world that do that.  Seems to be a minority.

> >> Certainly some more work can be done in this area, but I fear that
> the
> >> only way a real idea as to how much NAT444 really doe break things
> will
> >> be operational experience.
> >
> > Yep.  (Same as everything else.)
> >
> 
> I'm sure that will happen soon enough. I, for one, am not looking
> forward to the experience.

Neither am I.

But if major content providers cannot provide  on their
properties, and if ISPs and CPE vendors do not make IPv6
available and working, and if web browsers don't adopt faster
fallback to IPv4 when IPv6 is borked   We will all be 
behind NATs.

-d

RE: CGN fixed/hashed nat question

[Dan Wing]

> -Original Message-
> From: Eric Oosting [mailto:eric.oost...@gmail.com]
> Sent: Monday, January 21, 2013 9:06 AM
> To: NANOG
> Subject: CGN fixed/hashed nat question
> 
> Let me start out by saying I'm allergic to CGN, but I got to ask the
> question:
> 
> Some of the CGN providers are coming out with "fixed" nat solutions for
> their IPv6 transition/IPv4 preservation technologies to reduce logging.
> This appears to provide for a static mapping of outside ports/IPs to a
> particular customer such that the service provider doesn't need to log
> literally every session through the box.
> 
> At the last nanog, I seem to remember someone stepping up and discussing
> the problems associated with just taking ports 1025 through 1025+X and
> giving it to some customer and had brought up the idea of using a hash
> or salt to map what would appear to be random ports to a customer in
> such a way that you could reverse the port back to the customer later if
> need be.
> For the life of me, I can't find anything on the internets about this
> concept.
> 
> I had it in my head it was a lightning talk or something, but reviewing
> the agenda doesn't ring any bells. Anyone know what I'm talking about
> and what it's called?


later, Eric Oosting  wrote:

> > draft-donley-behave-deterministic-cgn
> 
> That's it. Or more specifically, the section of that draft that points
> to https://tools.ietf.org/html/rfc6431#section-2.2

I am also not a fan of CGN or NAT, but I co-chair the IETF's BEHAVE
working group that works on NAT.

draft-donley-behave-deterministic-cgn provides that functionality in
an attempt to help randomize ports (see RFC6056).  However, because
the ports are fixed and there are relatively few ports, an attacker
can determine the ports by causing the victim to open a bunch 
of TCP connections.  This can be done by a bunch of "img src" tags
in an HTML-encoded email message, among other mechanisms.  If the
hashing causes no logging, it creates a new requirement for a strong
audit trail of the CGN configuration.

The hashing provided by draft-donley-behave-deterministic-cgn is 
not necessary to avoid "logging every session".  Rather, the reduction
occurs by generating 1 logging event when creating  mapped
ports.  If using the CGN configuration, then no logging event needs
to be generated.

To date, the BEHAVE working group has not standardized any of
the proposed hashing techniques because several require coordination
between the devices (such as CPE and CGN), or between the log
generator and log consumer, or are functions self-contained within
a device and don't require standards action.

-d

RE: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6 naysayer...)

[Dan Wing]

> -Original Message-
> From: arin-ppml-boun...@arin.net [mailto:arin-ppml-boun...@arin.net] On
> Behalf Of Chris Grundemann
> Sent: Thursday, February 17, 2011 5:55 PM
> To: Benson Schliesser
> Cc: NANOG list; ARIN-PPML List
> Subject: Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6
> naysayer...)
> 
> On Thu, Feb 10, 2011 at 14:17, Benson Schliesser
>  wrote:
> 
> > If you have more experience (not including rumors) that suggests
> otherwise, I'd very much like to hear about it.  I'm open to the
> possibility that NAT444 breaks stuff - that feels right in my gut - but
> I haven't found any valid evidence of this.
> 
> In case you have not already found this:
> http://tools.ietf.org/html/draft-donley-nat444-impacts-01

That document conflates problems of NAT444 with problems of NAT44 
with problems of bandwidth starvation with problems of CGN.

For details, see my comments at
http://www.ietf.org/mail-archive/web/behave/current/msg09027.html
and see Reinaldo Penno's comments at
http://www.ietf.org/mail-archive/web/behave/current/msg09030.html

-d

> Cheers,
> ~Chris
> 
> >
> > Regardless, I think we can agree that IPv6 is the way to avoid NAT-
> related growing pains.  We've known this for a long time.
> >
> > Cheers,
> > -Benson
> >
> > ___
> > PPML
> > You are receiving this message because you are subscribed to
> > the ARIN Public Policy Mailing List (arin-p...@arin.net).
> > Unsubscribe or manage your mailing list subscription at:
> > http://lists.arin.net/mailman/listinfo/arin-ppml
> > Please contact i...@arin.net if you experience any issues.
> >
> 
> 
> 
> 
> 
> 
> --
> @ChrisGrundemann
> weblog.chrisgrundemann.com
> www.burningwiththebush.com
> www.theIPv6experts.net
> www.coisoc.org
> ___
> PPML
> You are receiving this message because you are subscribed to
> the ARIN Public Policy Mailing List (arin-p...@arin.net).
> Unsubscribe or manage your mailing list subscription at:
> http://lists.arin.net/mailman/listinfo/arin-ppml
> Please contact i...@arin.net if you experience any issues.

RE: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6 naysayer...)

[Dan Wing]

> -Original Message-
> From: Owen DeLong [mailto:o...@delong.com]
> Sent: Monday, February 21, 2011 12:59 PM
> To: Dan Wing
> Cc: 'Chris Grundemann'; 'Benson Schliesser'; 'NANOG list'; 'ARIN-PPML
> List'
> Subject: Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6
> naysayer...)
> 
> 
> On Feb 21, 2011, at 12:37 PM, Dan Wing wrote:
> 
> >> -Original Message-
> >> From: arin-ppml-boun...@arin.net [mailto:arin-ppml-boun...@arin.net]
> On
> >> Behalf Of Chris Grundemann
> >> Sent: Thursday, February 17, 2011 5:55 PM
> >> To: Benson Schliesser
> >> Cc: NANOG list; ARIN-PPML List
> >> Subject: Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6
> >> naysayer...)
> >>
> >> On Thu, Feb 10, 2011 at 14:17, Benson Schliesser
> >>  wrote:
> >>
> >>> If you have more experience (not including rumors) that suggests
> >> otherwise, I'd very much like to hear about it.  I'm open to the
> >> possibility that NAT444 breaks stuff - that feels right in my gut -
> but
> >> I haven't found any valid evidence of this.
> >>
> >> In case you have not already found this:
> >> http://tools.ietf.org/html/draft-donley-nat444-impacts-01
> >
> > That document conflates problems of NAT444 with problems of NAT44
> > with problems of bandwidth starvation with problems of CGN.
> >
> > For details, see my comments at
> > http://www.ietf.org/mail-archive/web/behave/current/msg09027.html
> > and see Reinaldo Penno's comments at
> > http://www.ietf.org/mail-archive/web/behave/current/msg09030.html
> >
> > -d
> 
> The document describes problems that will exist in NAT444 environments.
> It does not state that these problems would be specific to NAT444, but,
> NAT444 will cause or exacerbate each of the problems described.

To the contrary.

Its title, filename, abstract, and introduction all say the problems
are specific to NAT444.  Which is untrue.

> Yes, the problems may have other underlying root causes, but, they
> will all be present in a NAT444 environment, even if they were not
> present in the same environment prior to deployment of NAT444.
> 
> 
> Let me put it this way...
> 
> IPv4 has a TITANIC lack of numeric addresses and has been
> stretched beyond its limits for some time now.
> 
> IPv6 is a life boat.
> 
> NAT is a seat cushion used for floatation.
> 
> NAT444 (and other NAT-based extensions) are deck chairs.
> 
> Attempting to improve them beyond their current states is
> an effort to rearrange the deck chairs.

-d

RE: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6 naysayer...)

[Dan Wing]

> >> http://tools.ietf.org/html/draft-donley-nat444-impacts-01
> > That document conflates problems of NAT444 with problems of NAT44
> > with problems of bandwidth starvation with problems of CGN.
> 
> it may require a delicate palate to differentiate the different flavors
> of 

Running out of bandwidth for Netflix is pretty distinct from
the flavor of fried gNAT.

-d

RE: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6 naysayer...)

[Dan Wing]

> -Original Message-
> From: Chris Grundemann [mailto:cgrundem...@gmail.com]
> Sent: Monday, February 21, 2011 8:17 PM
> To: Dan Wing
> Cc: Owen DeLong; Benson Schliesser; NANOG list; ARIN-PPML List
> Subject: Re: [arin-ppml] NAT444 rumors (was Re: Looking for an IPv6
> naysayer...)
> 
> On Mon, Feb 21, 2011 at 19:08, Dan Wing  wrote:
> 
> > Its title, filename, abstract, and introduction all say the problems
> > are specific to NAT444.  Which is untrue.
> 
> I just re-read the filename, abstract and introduction, and I disagree
> that any of those say that the problems are specific to NAT444. They
> all do state that these problems are present in NAT444, but not that
> it's the only technology/scenario/configuration where you might find
> them.
> 
> More importantly, I am unsure the point of this argument.

My point is that:  NAT breaks things, but NAT444 is /not/ the only 
case where breakage occurs.

> Are you
> trying to say that the items listed as broken in the draft are not
> actually broken? Because in my experience they are. IMHO, the fact
> that they are also broken in other (similar) scenarios is not evidence
> that they are not broken in this one. On the contrary, this scenario
> seems to be evidence to the brokenness in the others (until we get a
> chance to test and document them all - are you volunteering? ;).

Vendor test results don't carry much value.

The authors of draft-donley-nat444-impacts did testing, and
I sincerely hope will publish results that split the impacts of
access bandwidth starvation from home NAT from CGN from NAT444.

-d


> Cheers,
> ~Chris
> 
> 
> > -d
> >
> >
> >
> 
> 
> 
> 
> --
> @ChrisGrundemann
> weblog.chrisgrundemann.com
> www.burningwiththebush.com
> www.theIPv6experts.net
> www.coisoc.org

RE: Yahoo and IPv6

[Dan Wing]

> -Original Message-
> From: George Bonser [mailto:gbon...@seven.com]
> Sent: Monday, May 16, 2011 2:10 AM
> To: Iljitsch van Beijnum; Owen DeLong
> Cc: NANOG list
> Subject: RE: Yahoo and IPv6
> 
> >
> > Because that way the IPv4 and IPv6 swarms remain disconnected in the
> > absence of some dual stack peers. (I.e., if the swarm is small and
> > you're the only IPv6 participant.)
> >
> > It would be much better if you could go from IPv6 to IPv4 through a
> > NAT64.
> 
> The problem is when the client is handed an explicit address rather
> than
> a host name.  In that case, there needs to be some standard environment
> variable for "IPv64 Prefix" that applications can query.
> 
> For a browser this might be something like the configured proxy.  Maybe
> an OS such as Windows might have a registry value for this.  Maybe
> Linux
> and other unix-like variations could have a sysctl for that.
> 
> There should be some standard way for a native v6 host to determine the
> v6 to v4 prefix to use in a NAT64 environment.

That need is acknowledged and being worked,
http://www.ietf.org/mail-archive/web/behave/current/msg09634.html

-d

AT&T postmaster

[🔓Dan Wing]

I help run a community machine (not for my employer) and we've been trying 
unsuccessfully for a month to get off AT&T's email blacklist; their automated 
systems claim we're off, but mail is still being blocked. Can an AT&T 
postmaster drop me an email to help work through this?

Thanks.
-d