Re: Level 3 Communications Issues Statement Concerning Comcast's Actions

[Jeff Wheeler]

On Mon, Nov 29, 2010 at 11:20 PM, Leo Bicknell  wrote:
> I will be the first to advocate the government use minimal to no
> regulation where there is active competition and consumer choice,
> and thus folks can "vote with their dollars".
>
> Broadband in the US is not in that boat.  Too many consumers have
> a "choice" of a single provider.  The vast majority of the rest
> have the "choice" of two providers.  We make these monopoly or

I believe regulation of peering among the largest networks in the U.S.
is a question of when and how, not if.  The more these incidents make
it into the news and attract the attention of public policy-makers,
the closer that "when" may become.  Comcast is either very clever, or
very stupid, for timing this in such a way that it has been spun into
an issue of who is streaming what into their customers' living rooms.

-- 
Jeff S Wheeler 

TWT - Comcast congestion

[Jeff Wheeler]

On Tue, Nov 30, 2010 at 9:12 PM, Richard A Steenbergen  
wrote:
> uncongested access. This is the kind of action that virtually BEGS for
> government involvement, which will probably end badly for all networks.

This depends on the eventual regulatory mechanism and the goals it
intends to promote.

Everyone in our industry has been aware that security mechanisms
related to BGP are needed, but after major incidents making it into
the news regularly for ten years,  little progress has been made.  A
regulator putting the hammer down might be a driving force to solve
some of our basically solvable problems that no one is willing to
spend any time or money on.

Additionally, it is easy to make the argument that reduced
interconnection cost for end-user ISPs would never motivate any
innovation.  If any network with 1000 DSL users could connect to the
closest PAIX (in every NFL city, of course) and gain access to all the
big players for nothing but the cost of transport, it would not
significantly reduce their cost to serve their customers.  The DSLAMs,
tech support monkeys, transport, idiotic implementation choices, etc.
cost an order of magnitude more than transit.  No regulator is going
to believe that eliminating the cost of transit will encourage more
broadband deployment, higher broadband speeds, or new inventions that
tax the network more heavily.

On the other hand, it is very easy for regulators to imagine that, if
Youtube had to bear the whole cost of moving bits from them to the
end-user, and broadband access was free for anyone with a house and
mailbox, developing new applications would be much more expensive and
happen less frequently.

I think eyeball networks had better start demonstrating how they are
innovating new things that benefit the public, and working hard to run
their networks and businesses efficiently, before the regulation
gauntlet is thrown down.  Otherwise, they will be on the losing end.
In either case, I don't think it automatically must be bad for all
networks, and everyone except those eyeball networks should hope it
turns out to be good for the public, increasing consumer choice and
bringing new forms of information and entertainment into their homes.

--
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: The scale of streaming video on the Internet.

[Jeff Wheeler]

On Thu, Dec 2, 2010 at 3:38 PM, Seth Mattinen  wrote:
> On 12/2/10 12:28 PM, Owen DeLong wrote:
>> You are assuming the absence of any of the following optimizations:
>>
>> 1.    Multicast
>
> Multicast is great for simulating old school broadcasting, but I don't
> see how it can apply to Netflix/Amazon style demand streaming where
I do.  Let's assume that there is a multicast future where it's being
legitimately used for live television, and whatever else.

The same mcast infrastructure will be utilized by Amazon.com to stream
popular titles (can you say New Releases) onto users' devices.  You
may be unicast for the first few minutes of the movie (if you really
want to start watching immediately) and change over to a
multicast-distributed stream once you have "caught up" to an
in-progress stream.

If Netflix had licensing agreements which made it possible for their
users to store movies on their local device, this would work even
better for Netflix, because of the "queue and watch later" nature of
their site and users.  I have a couple dozen movies in my instant
queue.  It may be weeks before I watch them all.  The most popular
movies can be multicast, and my DVR can listen to the stream when it
comes on, store it, and wait for me to watch it.

I am sure Amazon and Netflix have both thought of this already (if
not, they need to hire new people who still remember how pay-per-view
worked on C-band satellite) and are hoping multicast will one-day come
along and massively reduce their bandwidth consumption on the most
popular titles.  I am also certain the cable companies have thought of
it, and added it to the long list of reasons they will never offer
Internet multicast, or at least, not until a competitor pops up and
does it in such a way that customers understand it's a feature they
aren't getting.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Start accepting longer prefixes as IPv4 depletes?

[Jeff Wheeler]

How many networks already leak numerous unnecessary /24s to their
transit providers, who accept them (not having been asked to do
anything else), and contribute to table bloat?  Quite a lot of
networks do this.

Imagine if there are many possible inter-domain routes that are being
filtered by transit networks, because their customers accidentally
announce some number of /25-/32 networks to them.  These do not affect
us today; but I would hate to see all those accidental announcements
suddenly appear in my routing table; or for my transit providers to
have the bear the expense of dealing with them.

--
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Videotron contact

[Jeff Wheeler]

Could someone from Videotron contact me off-list?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

peering, derivatives, and big brother

[Jeff Wheeler]

A read through this New York Times article on derivatives clearing,
and the exclusivity that big banks seek to maintain, would look very
much like an article on large-scale peering, to someone who is not
expert in both topics.  The transit-free club and the "derivatives
dealers club" may have other similarities in the future, and it's
worth watching how further government regulation develops in this
area.  It may lead to insight into how government might eventually
regulate ISPs seeking to become settlement-free.

"“It appears that the membership criteria were set so that a certain
group of market participants could meet that, and everyone else would
have to jump through hoops,” Mr. Katz said."
http://www.nytimes.com/2010/12/12/business/12advantage.html?pagewanted=1&_r=1&src=busln

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: peering, derivatives, and big brother

[Jeff Wheeler]

Invisible Hand Networks was really meant to be a spot market.  The
same problem exists with bandwidth spot markets that always has
existed, the cost of ports to maintain sufficient capacity to the
exchange, and the lack of critical mass, meaning that the spot
bandwidth is either pretty expensive, or there is not enough capacity
for any serious application.  Certainly, no spot bandwidth market
currently in existence can compete with even mid-sized CDNs; and I do
not believe that will ever change.

The IHN folks were also disadvantaged because they seemed to know a
lot about economics, but basically nothing about networks.  So their
technology was neat from a reporting perspective, but the actual
functioning their exchange fabric was/is a disaster.

I do not know if they are still in business or if they are still
constrained by the flawed design they had in place several years ago.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

On Wed, Dec 15, 2010 at 2:52 PM, Ryan Finnesey
 wrote:
> I remember 5  years ago a company called Invisible Hand Networks that
> tried something like that.
>
> Cheers
> Ryan
>
>
> -Original Message-
> From: Laurent GUERBY [mailto:laur...@guerby.net]
> Sent: Monday, December 13, 2010 3:07 PM
> To: George Bonser
> Cc: nanog@nanog.org
> Subject: Re: peering, derivatives, and big brother
>
> On Sun, 2010-12-12 at 19:36 -0800, George Bonser wrote:
>> (...) The financial derivatives market isn't, in my opinion, a good
>> analogy of the peering market.  A data packet is "perishable" and must
>
>> be moved quickly.  The destination network wants the packet in order
>> to keep their customer happy and the originating network wants to get
>> it to that customer as quickly and cheaply as possible.  The
>> proliferation of these peering points means that today there is more
>> traffic going directly from content network to eyeball network.  To
>> use a different analogy, it is almost like the market is going to a
>> series of farmer's markets rather than supermarkets in the
>> distribution channel.  Sure, there are still the "supermarkets" out
>> there, but increasingly they are selling their "store brand" by
>> becoming content hosting networks themselves.  (...)
>
> Hi,
>
> The electricity spot market is close to your definition of "perishable":
>
> http://en.wikipedia.org/wiki/Electricity_market
>
> It has a derivative market, google for "electricity derivatives" will
> give you some papers and models.
>
> I'm pretty sure electricity and bandwidth share some patterns.
>
> Now who wants to be the Enron of the bandwidth market? :)
>
> Sincerely,
>
> Laurent
> http://guerby.org/blog
>

Re: Some truth about Comcast - WikiLeaks style

[Jeff Wheeler]

On Wed, Dec 15, 2010 at 5:47 PM, Adam Rothschild  wrote:
> I don't see how this point, however valid, should factor into the
> discussion.  Missing from this thread is that Comcast's topology and
> economics for hauling bits between a neutral collocation facility and
> broadband subscriber are the _same_ whether they ingest traffic by way
> of a settlement-free peer, customer, or paid transit connection.

Given that transit must be an incredibly small portion of Comcast's
cost to provide IP service to its customers, I think there are only
three possible reasons why Comcast would focus so much energy on
congesting transit to force content networks to purchase connectivity
for them, rather than upgrade transit or engage in more peering:

1) Comcast believes they can exact a great deal of revenue from
content networks.  For this to be comparable to their captive
customers, per-megabit rates must be reminiscent of pre-Level3 days,
when $30/Mb was a bargain.  This would spell bad news for Netflix.  Of
course, since cable companies typically must pay network affiliates
and media companies great sums for television programming packages, it
is in direct opposition to the TV content/delivery model.  It would be
hard to argue both sides if both businesses were faced with
like-minded regulators.

2) Comcast is making its engineering decisions in an ego-driven
manner, with little or no practical basis for their peering or transit
purchasing strategy.

3) Comcast is hoping the phrase "net neutrality" becomes a thing of
the past, and that, at some point in the future, they will be free to
block or QoS down anyone they please, including content networks,
search engines, or MP3 stores that compete with their own offerings.
I bet Comcast would love to have a few cents off every iTunes purchase
through their network, a handling charge for every amazon.com
transaction, or to coerce a million Netflix subscribers into a
Comcast-owned service.  This is as good a way as any for Comcast to
argue their side to potential regulators.

In any case, the "net neutrality" side gains credibility anytime a
media company can be made to look like they are constraining users'
choices by exacting a price from content providers.  There has been
talk of regulating Internet peering on this list since DIGEX
disconnected from ANS, if not before.  Reasons in favor of doing it
continue to become easier for a lay-person to understand.  In my
state, there is a law against walking down the street with an ice
cream cone in your pocket.  I don't know the origin of that law, but I
strongly suspect some person did it enough times, for a dumb enough
reason, to attract legislative interest.  Comcast should keep that in
mind before engaging in further peering brinkmanship.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Some truth about Comcast - WikiLeaks style

[Jeff Wheeler]

On Thu, Dec 16, 2010 at 12:15 PM, Dave Temkin  wrote:
> I disagree.  Even at $1/Mbit and 6Tbit of traffic (they do more), that's
> still $72M/year in revenue that they weren't recognizing before.  Given that
> that traffic was actually *costing* them money to absorb before, turning the
> balance and making that kind of money would be very favorably looked upon in

Yeah, because it makes a lot of sense to fuck with a billion dollar a
month revenue stream so you can extract a few million dollars more per
month from IP carriers.  This definitely makes more sense than, say,
running the billion dollar a month side a little more efficiently.

You need to understand the scale of comcast's expenses and revenue on
the access and transport side of their business, in order to have a
remotely intelligent opinion about whether or not they are doing
anything smart with the peering/transit side, in these conditions.

If you really think it's a good idea to attract the attention of
government regulators, newspapers, customers, and every major ISP by
making a lot of noise over something that might allow you to make 0.5%
more money off a product where you could probably save an order of
magnitude more money through any number of ignored efficiencies within
the organization, I would love for you to post that.  I suspect that
most folks who are of the opinion that Comcast is motivated by
anything but the three things I mentioned have not clearly considered
the proportionately small benefit they could gain from selling access
to their network at anything approaching a nominal fee.  It must be
either 1) very high per-Mb price; 2) ego and stupidity; 3) greed of
such magnitude that it would make Gordon Gecko proud.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Some truth about Comcast - WikiLeaks style

[Jeff Wheeler]

On Thu, Dec 16, 2010 at 1:53 PM, Dave Temkin  wrote:
> I do.  And yes, they are happy to "fuck with a billion dollar a month
> revenue stream" (that happens to be low margin) in order to set a precedent
> so that when traffic is 60Tbit instead of 6Tbit, across the *same* customer

We disagree on this point.  I do not think anyone knowledgeable at
Comcast realistically believes they will be able to charge a
business-relevant amount of money for access to their customers.  I
think regulators would first but the brakes on our whole industry.
Cable Internet is far from low-margin; in fact, the cable company in
my area, an order of magnitude smaller than Comcast, generates an
order of magnitude more profit from IP than from television.

What I do think, and what people on this list who engage in peering
discussions with Comcast cannot say for fear of reprisal, is that the
peering folks at Comcast are driven entirely by ego, and they lack the
big picture decision-making capacity of business people making
strategy decisions.  They have upper management convinced that
becoming settlement-free is a golden goose.  The peering folks would
be wise to reconsider their positions before upper management realizes
that their ego-driven staff are risking the golden goose they already
have, their captive audience, for little gain.  After all, if they
can't manage to run their IP and transport network more
cost-effectively than they do today, they will never be able to
compete as a transit network, and the golden goose they are chasing
will never lay any eggs.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Alacarte Cable and Geeks

[Jeff Wheeler]

On Fri, Dec 17, 2010 at 12:26 AM, Jay Ashworth  wrote:
> the 80s when that practice got started -- having to account for each
> individual subscriber pushed the complexity up, in much the same way
> that flat rate telecom services are popular equally because customers
> prefer them, and because the *cost of keeping track* becomes >delta.

Having personally and solely designed and written a toll billing
system from scratch that directly exchanged billing and settlement
data (and end-user data) with hundreds of ILECs, I can tell you a
number of things I learned:
1) billing is only as hard as you (or your vendor) make it
2) if your company can't figure out how to bill for a new product or
service, blame the billing people, not the product
3) keeping up with taxes and fees consume a lot more resources than
calculating the net bills themselves; so adding products is really
trivial compared to dealing with every pissant local government that
decides to apply a different taxing method to your HBO (or your
telephone calls)

This is not to say the folks that handle billing at cable companies
are equally capable, but if they had legitimate competitors, they
would figure out how to run many parts of their businesses more
efficiently.  Imagine if Wal-Mart was the only game in town that had
bar code readers at the cash registers, and every other grocery chain
had to look up every item and punch in the price to check you out.
Other stores would quickly improve their technology or find themselves
out of business.

> 2) New networks prefer it, and the fact that it happens makes the
> creation of new cable networks practical -- you don't have to go around
> and sell your idea to people retail; you sell it to CATV systems (well,

My understanding is that networks/media giants like it because they
can force cable companies to carry 11 irrelevant channels to get the
Disney Channel that your kids want.  Would enough people really ask
for G4TV to make producing and syndicating shows for that channel
cost-effective?  I don't know the answer, but my suspicion is that
people who really just want CSN, E!, or the Golf Channel are
subsidizing G4 viewers.  I wanted BBCA a few years ago, but my cable
provider required that I buy 30 other channels I did not want or had
never even heard of to get BBCA, so I didn't subscribe to it.

I do not know if a la carte channel selection would be good for me, as
a consumer, or not.  I do think the reasons the industry does not want
to offer that to end-users are disingenuous.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute

[Jeff Wheeler]

On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser
 wrote:
> I have no direct knowledge of the situation, but my guess:  I suspect the 
> proposal was along the lines of longest-path / best-exit routing by Level(3). 
>  In other words, if L(3) carries the traffic (most of the way) to the 
> customer, then Comcast has no complaint--the costs can be more fairly 
> distributed.  The "modest investment" is probably in tools to evaluate 
> traffic and routing metrics, to make this work.  This isn't really *new* to 
> the peering community, but it isn't normal either.

That is a reasonable guess, but Level3's FCC filing yesterday spells
out with certainty that Level3 did offer to "cold potato" traffic onto
Comcast (it does not mention the technical means e.g. MED honoring,
CDN smarts, or otherwise) and that Comcast refused.

I agree that the proposed Comcast solution may not be truly "new" but
instead unusual, but unless "Backdoor Santa" tells us what they really
have in mind, I suppose we won't know.  If I were Comcast, I would
want to move the significant cost of detailed netflow collection and
analysis infrastructure onto backbone providers by wrapping that
accounting mechanism up into my settlement agreements with peers, as
well as the expense of a cost-ineffective network, and demand that
Level3 and Comcast really calculate how much each network spends on
each bit, and share in that cost.  In theory, this is what happens
when an ILEC opens a rate case with its state regulator; and it is how
settlements for POTS calls work (at a very basic level.)

Actually, if I were Comcast, I would focus on running my business more
efficiently, as Level3 has thrown down the gauntlet with the FCC and
requested that the FCC dictate to Comcast specifically, and explicitly
all other broadband access providers, how they will interconnect with
peers and transit suppliers.  Level3 must think that their business
would be better off with regulatory oversight of peering, or they
would not have taken this action.  Comcast should realize that, of the
three potential motives for their recent actions I have previously
outlined, #1 and #3 are not just highly unlikely, but would be
practically impossible in a regulated environment.  As such, they
should further realize that their peering committee is driven by
motive #2, ego, and find the best way to change their position without
losing too much credibility.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: "potential new and different architectural approach" to solve the Comcast - L3 dispute

[Jeff Wheeler]

On Fri, Dec 17, 2010 at 12:48 PM, Richard A Steenbergen
 wrote:
> advertising MEDs, or by sending inconsistent routes. The fact that the
> existing Level3/Comcast routing DOESN'T make Level 3 haul all of the
> bits to the best exit mean it's highly likely that Comcast agreeing to
> haul the bits was part of their commercial transit agreement, probably
> in exchange for lower transit prices.

It's worth asking why Comcast did not accept Level3's suggestion that
they use MED as a face-saving maneuver, which would have allowed both
sides to declare victory.

A) Comcast may already have the contractual right to use MED but
chooses not to.  I agree with you that this is unlikely, not for pure
reasons of economics, but because Comcast has some of the same set of
motives not to send MED to their transit provider as every other
network: prefix aggregation, quality control, and ego.  I'll discount
geography, marketing, and inability to calculate useful MED values.
For argument's sake, let's say they currently can start sending MEDs
to Level3 whenever they want.  This being the case, Level3's "offer"
would have amounted to Level3 telling Comcast upper management that
Comcast's engineering people are leaving a huge amount of money on the
table, that Level3 is far more cost-effective at running its long-haul
network than Comcast, and that they should leave the big networking to
the big boys.  Comcast management could either react badly to this, or
go back to their network folks and ask why they can't be as
cost-effective as Level3.

B) Comcast may not be able to use MED today.  In this case, management
may be asking themselves why.  An essentially similar scenario can
play out; they can either react badly to Level3, or ask their own
staff why they are wasting money.

C) Comcast doesn't care about MED or the actual cost of doing
business.  They are boldly moving towards a future that is opposite
the one "net neutrality" folks advocate, one that looks like my
"Comcast Motive #3."

D) Comcast does not think that beginning to use MED (whether currently
enabled or not) is enough to satisfy the federal regulators and
legislators who are now taking interest in this game of
interconnection brinkmanship, involving 17 million households, between
a major IP carrier delivering traffic from everyone including a
household name like Netflix, and a major cable company that is waiting
for government approval to purchase NBC.  They feel they must demand
something very concrete to demonstrate that they are looking out for
consumers' best interest, which means they must make Level3 and/or
Netflix look like the bad guy.

E) Comcast thinks that a system of accounting for the cost of bearing
traffic and dividing it among the involved parties will actually be
good for their business, because they can over-build their
infrastructure as much as they like, perhaps even improving quality
for end-users, and only have to pay for about half of it.  The cost of
being inefficient, stupid, or committing purchasing or forecasting
errors drops by half.  This looks very much like my "Comcast Motive
#1."
E1) Comcast may also know a thing or two about Hollywood Accounting.
If you do not understand this reference, simply look it up on
Wikipedia.  It suffices to say that cost/revenue sharing agreements of
this nature can be manipulated in gross ways to the advantage of the
party doing the bulk of the book-keeping.

F) Management has the same case of ego-driven decision-making that
their technical staff have demonstrated.  I find this unlikely but
still possible.  We all know this has been the case at the CEO level
in some major interconnection disputes of the past.

I believe this outlines the reasonable scenarios for Comcast avoiding
a face-saving maneuver with Level3.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Some truth about Comcast - WikiLeaks style

[Jeff Wheeler]

On Sun, Dec 19, 2010 at 8:48 PM, Richard A Steenbergen  
wrote:
> Running a wire to everyone's house is a natural monopoly. It just
> doesn't make sense, financially or technically, to try and manage 50
> different companies all trying to install 50 different wires into every
> house just to have competition at the IP layer. It also wouldn't make

What no one has mentioned thus far is that CLECs really are able to
install their own facilities to homes and businesses if they decide
that is a good way to invest their finite resources.  This is why we
see several options for local loops in the "business district" of
every sizable city, as well as in many newly-developed areas such as
industrial parks.  These infrastructure builds are expensive, the
CLECs had limited logistical capabilities and could only manage so
many projects at once, and obviously, they focused their efforts on
the parts of town where return-on-investment was likely to be highest.
 Businesses often do have several good choices for voice, data,
Internet, and so on.  Cogent is an example of an essentially
Internet-only service having some degree of success at this without
even offering voice, or initially even transport, products.

The reason we will not see competitive facility builds to residences
is they have a very long ROI scale.  Everything in the traditional
telecommunications world did.  Many POTS customers still pay a fee for
DTMF or "touch tone dialing", because when their phone company
invested in new cards and software to support DTMF signalling, they
passed those expenses on to consumers.  These upgrades cost on the
order of a thousand dollars per phone line, but consumers could get
the benefit of DTMF by paying a couple dollars per month.  See also:
call waiting, caller ID, and so on.  I don't know about you, but I was
still using an "ATDP" dialing string until cable and DSL became
available to me at home (in about 2002) because I did not want to pay
the extra fee for touch tone dialing or other features I didn't need
on a dedicated modem line. ;)

We see examples of more choice available to business consumers than
residences, due to economies of scale, every day.  A business,
apartment community, or neighborhood association can choose from
multiple dumpster-tip services for trash collection.  Most residents
do not have enough trash volume to justify a bulky dumpster, so their
only practical choice is whatever curb-side trash collection company
has an agreement with their local government.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 BGP table size comparisons

[Jeff Wheeler]

I could not find this information on any Wikis, but this is the sort
of thing that would be nice to be able to find out without posting on
the list or asking around (obviously.)  I have quickly made a couple
of entries with simple enough formatting that anyone can go onto
Wikipedia, click Edit, and add what they know.  This is sure to become
a frequently asked question before the answer is always "yes" given
that some major transit-free networks have no functional IPv6
capability of any kind.

http://en.wikipedia.org/wiki/Comparison_of_IPv6_support_by_major_transit_providers

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 BGP table size comparisons

[Jeff Wheeler]

On Wed, Dec 22, 2010 at 2:24 AM, Pekka Savola  wrote:
> 'Maximum Prefix Length' may be an over-simplifying metric. FWIW, we're
> certainly not a major transit provider, but we do allow /48 in the
> designated PI ranges but not in the PA ranges.  So the question is not
> necessarily just about the prefix length used because it might vary by the
> prefix.

I know it is an over-simplification.  If someone wishes to edit the
page to provide more specific details about the route filtering policy
for a given transit network, Wikipedia is pretty easy to edit.
Hopefully they would provide a citation/link to the policy page for
the NSP as well.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Tue, Jan 4, 2011 at 11:35 PM, Kevin Oberman  wrote:
> The PDF is available at:

I notice that this document, in its nearly 200 pages, makes only
casual mention of ARP/NDP table overflow attacks, which may be among
the first real DoS challenges production IPv6 networks, and equipment
vendors, have to resolve.  Some platforms have far worse failure modes
than others when subjected to such an attack, and information on this
subject is not widely-available.

Unless operators press their vendors for information, and more knobs,
to deal with this problem, we may all be waiting for some group like
"Anonymous" to take advantage of this vulnerability in IPv6 networks
with large /64 subnets configured on LANs; at which point we may all
find ourselves scrambling to request knobs, or worse, redesigning and
renumbering our LANs.

RFC5157 does not touch on this topic at all, and that is the sole
reference I see in the NIST publication to scanning attacks.

I continue to believe that a heck of a lot of folks are missing the
boat on this issue, including some major equipment vendors.  It has
been pointed out to me that I should have been more vocal when IPv6
was still called IPng, but in 16 years, there has been nothing done
about this problem other than water-cooler talk.  I suspect that will
continue to be the case until those of us who have configured our
networks carefully are having a laugh at the networks who haven't.
However, until that time, it's also been pointed out to me that
customers will expect /64 LANs, and not offering it may put networks
at a competitive disadvantage.

Vendor solutions are needed before scanning IPv6 LANs becomes a
popular way to inconvenience (at best) or disable (at worst) service
providers and their customers.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 3:31 AM, Mohacsi Janos  wrote:
>        Do you have some methods in your mind to resolve ARP/ND overflow
> problem? I think limiting mac address per port on switches both efficient on
> IPv4 and IPv6. Equivalent of DHCP snooping and Dynamic ARP Inspection should
> be implemented by the switch vendors But remember DHCP snooping et al.
> implemented in IPv4 after the first serious attacks...Make pressure on your
> switch vendors

Equipment vendors, and most operators, seem to be silent on this
issue, not wishing to admit it is a serious problem, which would seem
to be a required step before addressing it.

Without more knobs on switches or routers, I believe there are only
two possible solutions for production networks today:
1) do not configure /64 LANs, and instead, configure smaller subnets
which will reduce the impact of scanning attacks
This is not desirable, as customers may be driven to expect a /64, or
even believe it is necessary for proper functioning.  I brought this
up with a colleague recently, who simply pointed to the RFC and said,
"that's the way you have to do it."  Unfortunately, configuring the
network the way the standard says, and accepting the potential DoS
consequences, will likely be less acceptable to customers than not
offering them /64 LAN subnets.  This is a foolish position and will
not last long once reality sets in, unless vendors provide more knobs.

2) use link-local addressing on LANs, and static addressing to end
hosts.  This prevents a subset of attacks originated from "the
Internet," by making it impossible for NDP to be initiated by scanning
activity; but again, is not what customers will expect.  It may have
operational disadvantages with broken user-space software, is not easy
for customers to configure, and does not permit easy porting of
addresses among host machines.  It requires much greater configuration
effort, is likely not possible by way of DHCP.  It also does not solve
NDP table overflow attacks initiated by a compromised host on the LAN,
which makes it a half-way solution.

The knobs/features required to somewhat-mitigate the impact of an NDP
table overflow attack are, at minimum:
* keep NDP/ARP entry alive based on normal traffic flow, do not expire
a host that is exchanging traffic
  + this is not the case with some major platforms, it surprised me to
learn who does not do this
  + may require data plane changes on some boxes to inform control
plane of on-going traffic from active addresses
* have configurable per-interface limits for NDP/ARP resource
consumption, to prevent attack on one interface/LAN from impacting all
interfaces on a router
  + basically no one has this capability
  + typically requires only control plane modifications
* have configurable minimum per-interface NDP/ARP resource reservation
  + typically requires only control plane modifications
* have per-interface policer for NDP/ARP traffic to prevent control
plane from becoming overwhelmed
  + because huge subnets may increase the frequency of scanning
attacks, and breaking one interface by reaching a policer limit is
much better than breaking the whole box if it runs out of CPU, or
breaking NDP/ARP function on the whole box if whole-box policer is
reached
* learn new ARP/NDP entry when new transit traffic comes from a host on the LAN
  + even if NDP function is impared on the LAN due to on-going scan attack
  + again, per-interface limitations must be honored to protect whole
box from breaking from one misconfigured / malicious LAN host
* have sane defaults for all and allow all to be modified as-needed

I am sure we can all agree that, as IPv6 deployment increases, many
unimagined security issues may appear and be dealt with.  This is one
that a lot of smart people agree is a serious design flaw in any IPv6
network where /64 LANs are used, and yet, vendors are not doing
anything about it.  If customers don't express this concern to them,
they won't do anything about it until it becomes a popular DoS attack.

In addition, if you design your network around /64 LANs, and
especially if you take misguided security-by-obscurity advice and
randomize your host addresses so they can't be found in a practical
time by scanning, you may have a very difficult time if the ultimate
solution to this must be to change the typical subnet size from /64 to
something that can fit within a practical NDP/ARP table.

Deploying /64 networks because customers demand it and your
competitors are doing it is understandable.  Doing it "because it's
the standard" is very stupid.  Anyone doing that on, for example,
SONET interfaces or point-to-point Ethernet VLANs in their
infrastructure, is making a bad mistake.  Doing it toward CE routers,
the sort that have an IPv4 /30, is even more foolish; and many major
ISPs already know this and are using small subnets such as /126 or
/124.

If you are still reading, but do not have any idea what I'm talking
about, ask yourself these questions

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum  wrote:
>> that a lot of smart people agree is a serious design flaw in any IPv6
>> network where /64 LANs are used
>
> It's not a design flaw, it's an implementation flaw. The same one that's in 
> ARP (or maybe RFC 894 wasn't published on april first by accident after all). 
> And the internet managed to survive.

It appears you want to have a semantic argument.  I could grant that,
and every point in my message would still stand.  However, given that
the necessary knobs to protect the network with /64 LANs do not exist
on any platform today, vendors are not talking about whether or not
they may in the future, and that no implementation with /64 LANs
connected to the Internet, or any other routed network which may have
malicious or compromised hosts, "design flaw" is correct.

This is a much smaller issue with IPv4 ARP, because routers generally
have very generous hardware ARP tables in comparison to the typical
size of an IPv4 subnet.  You seem to think the issue is generating NDP
NS.  While that is a part of the problem, even if a router can
generate NS at an unlimited rate (say, by implementing it in hardware)
it cannot store an unlimited number of entries.  The failure modes of
routers that have a full ARP or NDP table obviously vary, but it is
never a good thing.  In addition, the high-rate NS inquiries will be
received by some or all of the hosts on the LAN, consuming their
resources and potentially congesting the LAN.  Further, if the
router's NDP implementation depends on tracking the status of
"incomplete" on-going inquiries, the available resource for this can
very easily be used up, preventing the router from learning about new
neighbors (or worse.)  If it does not depend on that, and blindly
learns any entry heard from the LAN, then its NDP table can be totally
filled by any compromised / malicious host on the LAN, again, breaking
the router.  Either way is bad.

This is a fundamentally different and much larger problem than those
experienced with ARP precisely because the typical subnet size is now,
quite literally, seventy-quadrillion times as large as the typical
IPv4 subnet.

On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum  wrote:
> A (relatively) easy way to avoid this problem is to either use a stateful 
> firewall that only allows internally initiated sessions, or a filter that 
> lists only addresses that are known to be in use.

It would certainly be nice to have a stateful firewall on every single
LAN connection.  Were there high-speed, stateful firewalls in 1994?
Perhaps the IPng folks had this solution in mind, but left it out of
the standards process.  No doubt they all own stock in SonicWall and
are eagerly awaiting the day when "Anonymous" takes down a major ISP
every day with a simple attack that has been known to exist, but not
addressed, for many years.

You must also realize that the stateful firewall has the same problems
as the router.  It must include a list of allocated IPv6 addresses on
each subnet in order to be able to ignore other traffic.  While this
can certainly be accomplished, it would be much easier to simply list
those addresses in the router, which would avoid the expense of any
product typically called a "stateful firewall."  In either case, you
are now maintaining a list of valid addresses for every subnet on the
router, and disabling NDP for any other addresses.  I agree with you,
this knob should be offered by vendors in addition to my list of
possible vendor solutions.

On Wed, Jan 5, 2011 at 9:39 AM, Iljitsch van Beijnum  wrote:
> Sparse subnets in IPv6 are a feature, not a bug. They're not going to go away.

I do not conceptually disagree with sparse subnets.  With the
equipment limitations of today, they are a plan to fail.  Let's hope
that all vendors catch up to this before malicious people/groups.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 11:26 AM, Jon Lewis  wrote:
>> Anyone here use AltDB? It seems their servers have been down for two days.
> Can anyone from Level3 say how this will impact customer BGP filters. Will
> L3 keep working with the last data sync they got from altdb?  I'm guessing

Since Level3 updates their prefix-lists at least daily, and integrates
new ALTDB updates at least daily, and the ALTDB has been down for over
a day, obviously it will not affect your Level3 prefix-lists in the
near-term.  If Level3 decided to stop honoring ALTDB objects, say,
because ALTDB was never fixed, I imagine you would find it necessary
to re-publish your objects or Level3 would stop honoring your routes.

> I'd been thinking about moving from altdb to ARIN's but hadn't had
> sufficient motivation.

I emailed ARIN yesterday to ask if their IRR database has any
authentication support (other than mail-from) yet.  I haven't seen any
reply from ARIN yet, but my guess is they still have no useful
authentication mechanism.  I would rather depend on an IRR database
that can't process updates for a few days per year, than use one where
a malicious party could alter or erase all of my objects at any time.
I would like to note that RADB had route6: support in about 2004 or
so, if my memory serves me; while the ARIN database did not accept
route6 objects until about a year ago.  So it is not exactly a high
priority for ARIN.

Note also that Level3 has an IRR database, so you could use theirs if
you want to.  I don't prefer to use a transit provider database if I
can use a "neutral" one, but sometimes I would rather not pay the
(entirely reasonable) fee for the MERIT RADB.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 12:04 PM, Joel Jaeggli  wrote:
> no it isn't, if you've ever had your juniper router become unavailable
> because the arp policer caused it to start ignoring updates, or seen
> systems become unavailable due to an arp storm you'd know that you can
> abuse arp on a rather small subnet.

These conditions can only be triggered by malicious hosts on the LAN.
With IPv6, it can be triggered by scanning attacks originated from
"the Internet."  No misconfiguration or compromised machine on your
network is necessary.

This is why it is a fundamentally different, and much larger, problem.
 Since you seem confused about the basic nature of this issue, I will
explain it for you in more detail:

IPv4) I can scan your v4 subnet, let's say it's a /24, and your router
might send 250 ARP requests and may even add 250 "incomplete" entries
to its ARP table.  This is not a disaster for that LAN, or any others.
 No big deal.  I can also intentionally send a large amount of traffic
to unused v4 IPs on the LAN, which will be handled as unknown-unicast
and sent to all hosts on the LAN via broadcasting, but many boxes
already have knobs for this, as do many switches.  Not good, but also
does not affect any other interfaces on the router.

IPv6) I can scan your v6 /64 subnet, and your router will have to send
out NDP NS for every host I scan.  If it requires "incomplete" entries
in its table, I will use them all up, and NDP learning will be broken.
 Typically, this breaks not just on that interface, but on the entire
router.  This is much worse than the v4/ARP sitation.

I trust you will understand the depth of this problem once you realize
that no device has enough memory to prevent these attacks without
knobs that make various compromises available via configuration.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 12:26 PM, Phil Regnauld  wrote:
> Jeff Wheeler (jsw) writes:
>> Not good, but also does not affect any other interfaces on the router.
>        You're assuming that all routing devices have per-interface ARP tables.

No, Phil, I am assuming that the routing device has a larger ARP table
than 250 entries.  To be more correct, I am assuming that the routing
device has a large enough ARP table that any one subnet could go from
0 ARP entries to 100% ARP entries without using up all the remaining
ARP resources on the box.  This is usually true.  Further, routing
devices usually have enough ARP table space that every subnet attached
to them could be 100% full of active ARP entries without using up all
the ARP resources.  This is also often true.

To give some figures, a Cisco 3750 "pizza box" layer-3 switch has room
for several thousand ARP entries, and I have several with 3000 - 5000
active ARPs.  Most people probably do not have more than a /20 worth
of subnets for ARPing to a pizza box switch like this, but it does
basically work.

As we all know, a /64 is a lot more than a few thousand possible
addresses.  It is more addresses than anyone can store in memory, so
state information for "incomplete" can't be tracked in software
without creating problems there.  Being fully stateless for new
neighbor learning is possible and desirable, but a malicious host on
the LAN can badly impact the router.  This is why per-interface knobs
are badly needed.  The largest current routing devices have room for
about 100,000 ARP/NDP entries, which can be used up in a fraction of a
second with a gigabit of malicious traffic flow.  What happens after
that is the problem, and we need to tell our vendors what knobs we
want so we can "choose our own failure mode" and limit damage to one
interface/LAN.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 1:02 PM, TJ  wrote:
> Many would argue that the version of IP is irrelevant, if you are permitting
> external hosts the ability to scan your internal network in an unrestricted
> fashion (no stateful filtering or rate limiting) you have already lost, you

How do you propose to rate-limit this scanning traffic?  More router
knobs are needed.  This also does not solve problems with malicious
hosts on the LAN.

A stateful firewall on every router interface has been suggested
already on this thread.  It is unrealistic.

> Even granting that, for the sake of argument - it seems like it would not be
> hard for $vendor to have some sort of "emergency garbage collection"
> routines within their NDP implementations ... ?

How do you propose the router know what entries are "garbage" and
which are needed?  Eliminating active, "good" entries to allow for
more churn would make the problem much worse, not better.

-- 
Jeff S Wheeler  +1-212-981-0607
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Wed, Jan 5, 2011 at 8:57 PM, Joe Greco  wrote:
>> > This is a much smaller issue with IPv4 ARP, because routers generally
>> > have very generous hardware ARP tables in comparison to the typical
>> > size of an IPv4 subnet.
>>
>> no it isn't, if you've ever had your juniper router become unavailable
>> because the arp policer caused it to start ignoring updates, or seen
>> systems become unavailable due to an arp storm you'd know that you can
>> abuse arp on a rather small subnet.
>
> It may also be worth noting that "typical size of an IPv4 subnet" is
> a bit of a red herring; a v4 router that's responsible for /16 of
> directly attached /24's is still able to run into some serious issues.

It is uncommon for publicly-addressed LANs to be this large.  The
reason is simple: relatively few sites still have such an excess of
IPv4 addresses that they can use them in such a sparsely-populated
manner.  Those that do have had twenty years of operational experience
with generation after generation of hardware and software, and they
have had every opportunity to fully understand the problem (or
redesign the relevant portion of their network.)

In addition, there is not (any longer) a "standard," and a group of
mindless zealots, telling the world that at /16 on your LAN is the
only right way to do it.  This is, in fact, the case with IPv6
deployments, and will drive what customers demand.

To understand the problem, you must first realize that myopic
standards-bodies have created it, and either the standards must
change, operators must explain to their customers why they are not
following the standards, or equipment vendors must provide additional
knobs to provide a mitigation mechanism that is an acceptable
compromise.  Do the advantages of sparse subnets out-weigh the known
security detriments, even if good compromise-mechanisms are provided
by equipment vendors?

"Security by obscurity" is an oft-touted advantage of IPv6 sparse
subnets.  We all know that anyone with a paypal account can buy a list
of a few hundred million email addresses for next to nothing.  How
long until that is the case with lists of recently-active IPv6 hosts?
What portion of attack vectors really depend on scanning hosts that
aren't easily found in the DNS, as opposed to vectors depending on a
browser click, email attachment, or by simply hammering away at
"www.*.com" with common PHP script vulnerabilities?

How many people think that massively-sparse-subnets are going to save
them money?  Where will these cost-efficiencies come from?  Why can't
you gain that advantage by provisioning, say, 10 times as large a
subnet as you think you need, instead of seventy-quadrillion times as
large?  Is anyone really going to put their Windows Updates off and
save money because they are comfortable that their hosts can't be
found by random scanning?  Is stateless auto-configuration that big a
win vs DHCP?

Yes, I should have participated in the process in the 1990s.  However,
just because the bed is already made doesn't mean I am willing to lay
my customers in it.  These problems can still be fixed before IPv6 is
ubiquitous and mission-critical.  The easiest fix is to reset the /64
mentality which standards-zealots are clinging to.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 12:17 AM, Joe Greco  wrote:
> However, that's not the only potential use!  A client that initiates
> each new outbound connection from a different IP address is doing
> something Really Good.

No, Joe, it is not doing anything Good.  This would require the
software being written to make such random address selection, add more
entries to the router's NDP table, and it would DoS the box's own
router if an outbound scan were initiated from the host machine.
Again, you totally fail to understand the problem.  I should just
attach a "facepalm" graphic to my reply and stop bothering with your
idiocy, but it is important that as many people as possible understand
these issues.  Every additional person who is expressing concern to
their vendors brings us closer to a solution.

In addition, if the machine can choose another random IPv6 address in
the /64 for each new outgoing connection, it could just as easily
source all its outgoing connections from ... a single IPv6 address
that does not accept any incoming connections.  I will grant that this
does not prevent "magic packet" attacks against bugs in the machine in
kernel space, and that the machine could be the recipient of a DoS
attack; but your proposed solution has no advantage in the case of,
say, SQL Slammer 6, or other attacks exploiting vulnerabilities in
user-space.

On Thu, Jan 6, 2011 at 12:25 AM, Owen DeLong  wrote:
> Is there any reason we really need to care what size other people use for 
> their Point to Point
> links?
>
> Personally, I think /64 works just fine.
>
> I won't criticize anyone for using it. It's what I choose to use.

You should care what size you choose, Owen.  I would if I was your
customer.  There are several reasons why.  If you configure a /64 on a
point-to-point interface e.g. SONET, some current platforms will
bounce any "scan" packets back and forth between the two hosts until
the TTL expires, which means an attacker can saturate the
point-to-point link with two orders of magnitude less attack traffic
than the link capacity.  This is very bad.

Also, if the link is a LAN, you are vulnerable to the ARP/NDP problem.
 This may be a per-interface issue on your box, or it may be a global
one.  In the per-interface case, most likely, the failure mode will be
okay; but some vendors (including Juniper?) will eventually expire the
ARP/NDP entry even though there is active traffic, and may be unable
to re-learn it when needed due to the attack, which would break the
link between routers.  On the other hand, if it's a global issue, it
will break NDP on the entire router, affecting all interfaces by
whatever the platform's failure mode is.  Obviously, that is worse.

This is why "RFC compliant" is wrong, Owen.  That a learned,
experienced person such as yourself has no clue what the underlying
problem is exemplifies my point: much, much more noise needs to be
made about this issue.  A lot of smart people have known about it for
years but nothing is being done.

Anyone can choose not to use /64 on their point-to-point links with no
consequence to their business.  Customers aren't going to choose
another vendor because they are likely never even aware of it.  The
operational problem is that customers will want it on the PE/CE LANs,
hosting center LANs, and so on; and right now, if you tell them "we
can't do that," they have trouble believing you because the standard
says otherwise, and a lot of other networks are currently doing it
(because they think they don't have a choice other than to lose
potential customers.)

However, if you are running /64 instead of something like /124 on your
VLANs or SONET circuits between routers, you should change this
practice as soon as practical.  Not doing so once you have been
educated about this problem because it is "the standard" is very
stupid.  Pretty much every major transit network has figured this out
and are doing it on their interfaces to BGP customers, too, either
optionally, by default, or as the only choice.

The industry standard must change to be non-hostile toward choosing a
smaller subnet size than /64 to give coverage to those of us who do
understand what we are doing as we roll out IPv6 to customers.

--
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 12:54 AM, Joe Greco  wrote:
> I'm starting off with the assumption that knowledge of the host
> address *might* be something of value.  If it isn't, no harm done.
> If it is, and the address becomes virtually impossible to find, then
> we've just defeated an attack, and it's hard to see that as anything
> but positive.

I'm starting off with the assumption that the layer-3 network needs to
function for the host machines to be useful.  Your position is to just
hand any attacker an "off switch" and let them disable any (LAN |
router, depending on router failure mode) for which they know the
subnet exists, whether or not they know any of its host addresses.

This is a little like spending money on man-traps and security guards,
but running all your fiber through obvious ducts in a public parking
garage.  It may be hard to compromise the hosts, but taking them
offline is trivial.

On Thu, Jan 6, 2011 at 1:01 AM, Kevin Oberman  wrote:
> I am amazed at the number of folks who seem to think that there is time to
> change IPv6 is ANY significant way. Indeed, the ship has failed. If you
> r network is not well along in getting ready for IPv6, you are probably
> well on you way to being out of business.

There are many things that can change very easily.  Vendors can add
knobs, subnet size can get smaller (it works just fine today, it just
isn't "standard"), and so on.  A TCP session today looks a lot
different than it did in the mid-90s.  Now we have things like SYN
cookie, window scaling, we even went through the "hurry up and
configure TCP MD5 on your BGP just in case."  Fixing this problem by
deploying subnets as a /120 instead of a /64 is a lot easier than any
of those changes to TCP, which all required operating system
modifications on one or both sides.  How many networks honor ICMP
route-record, source routing, or make productive use of redirects (if
they have not outright disabled it?)  How many networks decided to
block all ICMP traffic because some clueless employee told them it was
smart?  CIDR routing?  Do you recall that the TTL field in IP headers
was originally not a remaining-hops-count, but actually, a value in
seconds (hence "Time To Live")?  IPv4, and the things built on top of
it, have evolved tremendously, some, all the way to the host network.

A lot of this evolution took place before it was common to conduct a
credit card transaction over the Internet, at a time when it really
was not mission-critical for most operators.  IPv6 is still not there,
but I agree, we are rapidly approaching that time, and much more than
90% of IPv4 networks have a lot of work to do.  It would be good to
see LANs smaller than /64 accepted sometime before IPv6 does become
widely-deployed to end users.

Or some other practical solution to the problems of huge subnet sizes,
whatever those solutions may be.  My guess is there may be other, very
significant, challenges to having huge LAN subnets.  This is one we
actually know about, but are choosing not to solve.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 2:42 AM, Joel Jaeggli  wrote:
> icmp6 rate limiting both reciept and origination is not rocket science.
> The attack that's being described wasn't exactly dreamed up last week,
> is as observed not unique to ipv6, and can be mitigated.

That does not solve the problem.  Your response, like most on this
thread, clearly indicates that you do not understand the underlying
issue, or how traffic is actually forwarded.  Neither IPv6 or IPv4
packets are simply forwarded onto the Ethernet, which is why the
ARP/NDP table resource is required; a mapping from layer-3 to layer-2
address is needed.  If the table resource for these entries is
exhausted, no new mappings can be learned, and bad things happen.
Either hosts on the specif interface, or the entire box, can no longer
exchange traffic through the router.  If an artificial rate-limit on
discovery traffic is reached, discovery of mappings will also be
impeded, meaning the denial-of-service condition exists and persists
until the attack ceases.  This may also affect either just that
interface, or all interfaces on the router, depending on its failure
mode.

Rate-limiting discovery traffic still breaks the attached LANs.  How
badly it breaks them is implementation-specific.  It does avoid using
up all the router's CPU, but that doesn't help the hosts which can't
exchange traffic.  Again, depending on the router implementation, the
fraction of hosts which cannot exchange traffic may reach 100%, and in
effect, the router might as well be down.

> You can still have this problem when you assign a bunch of /112s how
> many neighbor unreachable entries per interface can your fib hold?

You are correct, but the device can hold a significant number of
entries compared to the size of a /112 subnet, just like it can hold a
significant number of v4 ARP entries compared to a v4 /22 subnet.  The
difference is, ARP/NDP mappings for one /64 subnet can fill all the
TCAM resources of any router that will ever exist.  This is why more
knobs are needed, and until we have that, the /64 approach is
fundamentally broken.

Again, until this problem is better-understood, it will not be solved.
 Right now, there are many vulnerable networks; and in some platforms
running a dual-stack configuration, filling up the v6 NDP table will
also impact v4 ARP.  This means the problem is not confined to a cute
beta-test that your customers are just starting to ask about; it will
also take out your production v4 network.  If you are running a
dual-stack network with /64 LANs, you had better start planning.  It's
not just a problem on the horizon, it's a problem right now for many
early-adopters.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 4:32 AM, Joel Jaeggli  wrote:
> Which at a minimum is why you want to police the number of nd messages
> that the device sends and unreachable entries do not simply fill up the
> nd cache, such that new mappings in fact can be learned because there

Your solution is to break the router (or subnet) with a policer,
instead of breaking it with a full table.  That is not better; both
result in a broken subnet or router.  If NDP requires an NDCache with
"incomplete" entries to learn new adjacencies, then preventing it from
filling up will ... prevent it from learning new adjacencies.  Do you
see how this is not a solution?

> are free entries. you need to do this on a per subnet basis rather than
> globally. as in ipv4 policing, global limits will kill the boxes ability
> to learn new entries everywhere.

Yes, per-subnet/interface limits are absolutely needed, to prevent the
entire box from being killed when one subnet/interface becomes
impaired.  That won't stop attackers from breaking your network, both
because one subnet that presumably has production services on it is
broken, and because, presumably, the attacker can identify other
subnets on the router.  Even if not, the problem remains that ... some
subnets/interfaces are broken.

On Thu, Jan 6, 2011 at 5:20 AM, Owen DeLong  wrote:
>> You must also realize that the stateful firewall has the same problems
> Uh, not exactly...

Of course it does.  The stateful firewall must either 1) be vulnerable
to the same form of NDP attack; or 2) have a list of allocated v6
addresses on the LAN.  The reason is simple; a "stateful firewall" is
no more able to store a 2**64 table than is a "router."  Calling it
something different doesn't change the math.  If you choose to solve
the problem by disabling NDP or allowing NS only for a list of "valid"
addresses on the subnet, this can be done by a stateless router just
like on a stateful firewall.

> Uh, no it doesn't. It just needs a list of the hosts which are permitted
> to receive inbound connections from the outside. That's the whole

This solution falls apart as soon as there is a compromised host on
the LAN, in which case the firewall (or router) NDP table can again be
filled completely by that compromised/malicious host.  In addition,
the "stateful firewall," by virtue of having connection state, does
not solve the inbound NDP attack issue.  The list of hosts which can
result in an NDP NS is whats causes this, and such a list may be
present in a stateless router; but in both cases, it needs to be
configured.

"Stateful firewall" is unfortunately not magic dust that you can
sprinkle on any network problem.

> Except that routers don't (usually) have the ability to do dynamic outbound
> filtration which means that you have the scaling problem you've described
> of having to list every host on the net. If the router does have this ability,
> then, the router is, by definition, a stateful firewall.

As I state above, this capability does not "fix" the problem because
the "stateful firewall" can just as easily be subject to DoS.  You
must have a list of valid v6 hosts on the subnet for your solution to
work.

> There are risks with sparse subnets that have been inadequately addressed
> for some of their failure modes at this time. I wouldn't go so far as saying 
> they
> are a plan to fail. In most cases, most networks shouldn't be susceptible
> to an externally initiated ND attack in the first place because those should
> mostly be blocked at your border except for hosts that provide services
> to the larger internet.

As you have pointed out, without state information, you can't block
this external attack.  Even if you have a "stateful firewall," that
firewall must itself have a solution for the internally-originated NDP
attack.  While the problems are slightly different and the
internally-originated attack is less likely, both must be solved to
ensure a reliable v6 network.  The "stateful firewall" only solves
half the problem and remains entirely vulnerable to the other half.

On Thu, Jan 6, 2011 at 5:29 AM, Owen DeLong  wrote:
> But, Jeff, if the router has a bunch of /24s attached to it and you scan
> them all, the problem is much larger than 250 arp entries.

That depends on how many is "a bunch" and how large is the ARP table
on the device.  A pizza box layer-3 switch, as I have mentioned,
easily holds several thousand entries, modern units upwards of 10,000.
 That's enough room for several v4 /24 networks.  No device has enough
for one v6 /64 network.  In addition, most of the IPs on your typical
/24 networks will be in use routinely (in a hosting environment,
anyway) which means that a scan really doesn't generate many new
WHO-HAS and incomplete entries, because most layer-2 mappings are
already known.  Those that aren't fit within the cheap router's
resource limitations somewhat easily.

On Thu, Jan 6, 2011 at 5:41 AM, Phil Regnauld  wrote:
>        Additionnally I believe the size of ty

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 7:34 AM, Robert E. Seastrom  wrote:
> I continue to believe that the "allocate the /64, configure the /127
> as a workaround for the router vendors' unevolved designs" approach,

As a point of information, I notice that Level3 has deployed without
doing this, e.g. they have densely packed /126 subnets which are
conceptually identical to /30s for infrastructure and point-to-point.
I am still taking your conservative approach, as I see no operational
disadvantage to it; but opinions must differ.

On Thu, Jan 6, 2011 at 11:28 AM,   wrote:
> And the "ZOMG they can overflow the ARP/ND/whatever table" is a total red
> herring - you know damned well that if a script kiddie with a 10K node botnet
> wants to hose down your network, you're going to be looking at a DDoS, and it
> really doesn't matter whether it's SYN packets, or ND traffic, or forged ICMP
> echo-reply mobygrams.

I agree, botnets are large enough to DDoS most things.  However, with
the current state of ARP/ND table implementations in some major
equipment vendors' routers, combined with standards-compliant
configuration, it doesn't take a botnet.  I could DoS your subnet (or
whole router) without a botnet, say, using an IPv6 McDonald's Wi-Fi
hotspot.  That is very much a legitimate concern.  A few hundred
packets per second will be enough to cripple some platforms.

On Thu, Jan 6, 2011 at 1:20 PM, Owen DeLong  wrote:
> And there are ways to mitigate ND attacks as well.

No, Owen, there aren't.  The necessary router knobs do not exist.  The
"Cisco approach" is currently to police NDP on a per-interface basis
(either with per-int or global configuration knob) and break NDP on
the interface once that policer is exceeded.  This is good (thanks,
Cisco) because it limits damage to one subnet; but bad because it
exemplifies the severity of the issue: the "Cisco solution" is known
to be bad, but is less bad than letting the whole box break.  Cisco is
not going to come up with a magic knob because there isn't any -- with
the current design, you have to pick your failure modes and live with
them.  That's not good and it is not a Cisco failing by any means, it
is a design failing brought on by the standards bodies.


I would also like to reply in public to a couple of off-list
questions, because the questions are common ones, the answers are not
necessarily cut-and-dry (my opinion is only one approach; there are
others) and this is the kind of useful discussion needed to address
this matter.  I will leave out the names of the people asking since
they emailed me in private, but I'm sure they won't mind me pasting
their questions.

Anonymous Questioner:
>What do you think about using only link-local ip addresses on the 
>infrastructure links please?
I can't think of any potential drawbacks do you?

This can be done, but then you don't have a numbered next-hop for
routing protocols.  That's okay if you re-write it to something else.
Note that link-local subnets still have an NDP table, and if that
resource is exhausted due to attacks on the router, neighbors with
link-local addressing are not immune.

link-local scope offers numerous advantages which are mostly
out-weighed by more practical concerns, like, how hard it is going to
be to convince the average Windows sysadmin to configure his machine
to suit such a design, instead of just taking his business elsewhere.
Not a problem for enterprise/gov/academia so much, but a problem for
service providers.

On Thu, Jan 6, 2011 at 3:43 PM, Jack Bates  wrote:
> Given that the incomplete age is to protect the L2 network from excessive
> broadcast/multicast, I agree that aging them out fast would be a wiser

I agree that it would be nice to have such a knob.  I bet you and I
would make different choices about how to use it, which is the whole
point of having a knob instead of a fixed value.

> I'm still a proponent for removing as needed requests like this, though. It
> would have been better to send a global "everyone update me" request
> periodically, even if triggered by an unknown entry, yet limited to only
> broadcasting once every 10-30 seconds.

> Given that all requests for an unknown arp/ND entry results in all hosts on
> the network checking, it only makes sense for all hosts to respond. There

This isn't a new idea and it has its own set of problems, which are
well-understood.  IPv6 NS messages are more clever than ARP, though,
and are sent to a computed multicast address.  This means that the
number of hosts which receive the message is minimized.  See RFC2461
section 2.3 for the quick introduction.

NDP is better than ARP.  However, your statement that NDP has all (I'd
like to say some) of the same problems as ARP but the increased subnet
size has magnified them, is basically correct.  NDP does some things a
lot better than ARP, but not this.  It's important to realize that
when this stuff was designed, there were few hardware-based layer-3
routers for IPv4.  The biggest networks (Sprin

Re: IPv6 - real vs theoretical problems

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 5:00 PM, Deepak Jain  wrote:
> As far as I can tell, this "crippling" of the address space is completely 
> reversible, it's a reasonable step forward and the only "operational" loss is 
> you can't do all the address jumping and obfuscation people like to talk 
> about... which I'm not sure is a loss.

I largely agree with you, but my knowledge is similar to yours, and
does not extend to dealing with low-end CPE, dormitory LANs, hot
spots, or mobile networks.  I am by no means an authority in those
areas and I remain open to the possibility that there may be some
operational advantages to the IPv6 addressing concept for those users.
 The problem is there are very serious operational disadvantages for
you and I, but the standard tells us to do it anyway.  I would like
the hot spot or mobile guys to be able to choose /64 if they want to.
I need to choose otherwise, and customers expecting /64 as the
"standard" are going to be disappointed until the standard allows for
different choices.

I don't have an opinion on whether the address space is truly
"crippled."  If I did, I'm not sure it would be useful.  Classful
addressing ran out of gas in IPv4, so IPv6 has a huge number of bits
to hopefully avoid a repeat of that.  Okay, I can buy into that.
There are some major networks who aren't, though, and I think they
made a very conscious choice.  We won't know if it's a necessary
choice for a long time.

I will choose to devote my arguing-on-the-mailing-list time to topics
I think are more useful to discuss.  I do not think you will change
very many minds about IPv6 numbering schemes.  I hope I will change
some minds about the current safety of public /64 LANs and get more
folks talking to their vendors about it, which should give us some
kind of solution sooner, rather than later.  Choose your battles.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 6:46 PM, Owen DeLong  wrote:
> On Jan 5, 2011, at 9:17 PM, Joe Greco wrote:
>> However, that's not the only potential use!  A client that initiates
>> each new outbound connection from a different IP address is doing
>> something Really Good.
> If hosts start cycling their addresses that frequently, don't you run the
> risk of that becoming a form of DOS on your router's ND tables?

Of course, Owen.  I replied to that specific point in Joe's post
earlier, although I have written so much on this thread, I have tried
to condense my replies, so anyone reading in thread mode may have
missed it.

The fact that Joe even makes that suggestion signals how little
understanding he has of the problem.  His idea would DoS his own
router.  There are many posts on this thread from folks who think of
themselves as expert, at least enough to try and tell me that I'm
wrong, when they lack basic understanding of how the forwarding
process works in operation.  That is what everyone should be afraid of
-- most of the "experts" aren't, and almost no one has practical
experience with a mission-critical IPv6 network, so conditions like
this remain unanalyzed.  It took a long time to discover a lot of
vulnerabilities as the Internet grew from academia to everyday
necessity.  We are all now making some obvious, unnecessary mistakes
with IPv6 deployments.

It is also crucial to understand that some platforms use the same
resources (in control plane or data plane) for ARP and NDP tables and
resolution, and this means that some dual-stack networks will see
their IPv4 networks melt down due to problems with their IPv6 network
design and implementation.  If you are dual-stack, this is probably
not a problem confined to v6 traffic flowing through your network; it
may also take out your mission-critical v4 services.  If you don't
know, then you need to admit you don't know and find out what the
failure mode of your routers is, before your network blows up in your
face.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 - real vs theoretical problems

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 8:04 PM, Jimmy Hess  wrote:
> It is advisable to look for much stronger reasons than "With
> IPv4 we did it"  or   With IPv4 we ran into such and such
> problem"   due to unique characteristics of IPv4 addressing
> or other IPv4 conventions that had to continue to exist for
> compatibility reasons, etc, etc.

I certainly agree that there are many advantages to the greater
address space offered by IPv6.  I don't mean to advocate that we do
things the same way as was necessary to conserve v4 address space, and
I'm sure we all realize that RIR policies necessarily contributed to
routing table growth in trade for extending the life of the available
address space.

I'm not blindly deploying /64 networks, either.  Doing so with the
current set of problems, and lack of knobs, is very foolish.  My
transit providers offer a mix of /126 and /124 demarc subnets so far,
and /124 is what I choose to standardize on for my BGP customers and
private peering, for simplicity and convenience.  As I mentioned
before, I currently allocate a /64 and configure a /124, so I am not
painting myself into a corner either way.

How many of us with an appreciable level of expertise remain concerned
that our approach may need significant adjustment?  How many think we
know what those potential adjustments may be, and have planned to make
them easy (or transparent) for ourselves and customers if they become
necessary?  This is what is IMO most important to a responsible IPv6
deployment.  To do otherwise is inviting unpredictable future pain.

I am comforted by the fact that Level3 is deploying customer demarc
subnets as /126 and is NOT allocating a /64 for each, but are instead
packing them densely in an IPv4 /30 fashion.  They recognize problems
with the /64 approach, choose not to follow the "standard" to the
letter, and implement their dual-stack network in a way they
presumably believe is safe and scalable.  Large networks like Level3
choosing to insist that equipment vendors support this configuration,
rather than have problems with densely packed subnets smaller than
/64, will mean that anyone who wants to sell a router to Level3 had
better make it work correctly this way, which is good for the small
guy like me who thinks he will eventually transition to that
configuration.  Right now, I am still hedging my bet.

Are there any large transit networks doing /64 on point-to-point
networks to BGP customers?  Who are they?  What steps have they taken
to eliminate problems, if any?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 8:47 PM, Owen DeLong  wrote:
> 1.      Block packets destined for your point-to-point links at your
>        borders. There's no legitimate reason someone should be

Most networks do not do this today.  Whether or not that is wise is
questionable, but I don't think those networks want NDP to be the
reason they choose to make this change.

> 2.      For networks that aren't intended to receive inbound requests
>        from the outside, limit such requests to the live hosts that
>        exist on those networks with a stateful firewall.

Again, this doesn't fix the problem of misconfigured hosts on the LAN.
 I can and shall say it over and over, as long as folks continue to
miss the potential for one compromised machine to disable the entire
LAN, and in many cases, the entire router.  It is bad that NDP table
overflow can be triggered externally, but even if you solve that
problem (which again does not require a stateful firewall, why do you
keep saying this?) you still haven't made sure one host machine can't
disable the LAN/router.

> 3.      Police the ND issue rate on the router. Yes, it means that
>        an ND attack could prevent some legitimate ND requests
>        from getting through, but, at least it prevents ND overflow and
>        the working hosts with existing ND entries continue to function.
>        In most cases, this will be virtually all of the active hosts on
>        the network.

You must understand that policing will not stop the NDCache from
becoming full almost instantly under an attack.  Since the largest
existing routers have about 100k entries at most, an attack can fill
that up in *one second.*

On some platforms, existing entries must be periodically refreshed by
actual ARP/NDP exchange.  If they are not refreshed, the entries go
away, and traffic stops flowing.  This is extremely bad because it can
break even hosts with constant traffic flows, such as a server or
infrastructure link to a neighboring router.  Depending on the attack
PPS and policer configuration, such hosts may remain broken for the
duration of the attack.

Implementations differ greatly in this regard.  All of them break
under an attack.  Every single current implementation breaks in some
fashion.  It's just a question of how bad.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 9:31 PM, Owen DeLong  wrote:
>> You must understand that policing will not stop the NDCache from
>> becoming full almost instantly under an attack.  Since the largest
>> existing routers have about 100k entries at most, an attack can fill
>> that up in *one second.*
>>
> If the policing rate is set to ~100 requests per second, or, even
> 1000 requests per second, then, I'm not sure why you think this.

With a 100pps policer, it is trivial for an attack to make its NS
requests far more likely to make it through the policer than
legitimate NS requests that would result in discovering a valid
layer-2 mapping.  If you are hitting the policer, the subnet is
broken.  If you don't have a policer, the table is full and ... the
subnet is broken.  See how it's a problem that isn't solvable with a
simple policer?  Note that the Cisco "solution" is indeed a
configurable per-interface policer, which is better than nothing, but
does not fully solve the problem.  Policing isn't a new idea.  I'm not
sure it's a step in the right direction, or just prolonging an
inevitable change towards a real fix.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: NIST IPv6 document

[Jeff Wheeler]

On Thu, Jan 6, 2011 at 9:24 PM, Joe Greco  wrote:
> With today's implementations of things?  Perhaps.  However, you
> show yourself equally incapable of grasping the real problem by
> looking at the broader picture, and recognizing that problematic
> issues such as finding hosts on a network are very solvable
> problems, and that we are at an early enough phase of IPv6 that
> we can even expect some experiments will be tried.
>
> Look beyond what _is_ today and see if you can figure out what
> it _could_ be.  There's no need for what I suggest to DoS a router;
> that's just accepting a naive implementation and saying "well this
> can't be done because this one way of doing it breaks things."  It
> is better to look for a way to fix the problem.

Actually, unlike most posters on this subject, I have a very good
understanding of how everything works "under the hood."  For this
reason, I also understand what is possible given the size of a /64
subnet and the knowledge that we will never have adjacency tables
approaching this size.

If you are someone who thinks, oh, those Cisco and Juniper developers
will figure this out, they just haven't thought about it hard enough
yet, I can understand why you believe that a simple fix like "no ip
directed-broadcast" is on the horizon.  Unfortunately, it is not.  The
only thing they can do is give more mitigation knobs to allow
operators to choose our failure modes and thresholds.  To really fix
it, you need a smaller subnet or a radical protocol change that will
introduce a different set of problems.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Sat, Jan 8, 2011 at 2:47 PM, Christopher Morrow
 wrote:
> I don't think rr.arin.net and RPKI have anything to do with each
> other. I think the direction the RPKI should/is taking is to have the

I at least think that whatever future and time-table is planned for
RPKI, this should not stand in the way of ARIN offering an effective
authentication mechanism for the ARIN IRR.  FYI, the reply I received
from ARIN was that there are no plans to improve its authentication
capability.  I didn't ask why and don't really care why it has never
had anything more than MAIL-FROM in the past.  Either it should be
improved (IMO) or it shouldn't be.

I really do wonder what ARIN's plan is if a bad guy decides to forge
emails and delete or modify some or all of the objects.  Would they
just shut it down, improve authentication, or keep doing business as
usual?  I am always surprised that black hat folks do not do things
like this when faced with a damaging vulnerability that can easily be
exploited with no way to trace the activity back to the bad guy.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Sat, Jan 8, 2011 at 10:23 PM, Randy Bush  wrote:
> but, unlike the other regions, the arin.irr is not confuddled with the
> arin.whois.  i.e. it is kind of irrelevant to the authority on resource
> ownership, arin's real responsibility.

I certainly agree with this, and I am admittedly ignorant of the
history here, but I don't understand why ARIN is operating an IRR that
is very much insecure, instead of just not operating one at all.

> they are just providing a free irr service, as it is the popular thing
> for rirs to do these years.  and i don't think many use it.  if you

In terms of database size, excluding RIPE, the ARIN IRR is the 8th
largest, ahead of ALTDB and about 10% as large as Level3, the second
largest IRR database (except RIPE.)  A mass-corruption of the ARIN IRR
overnight might be a serious incident causing service impact to a
large number of users and businesses, and cause probably thousands of
people to be got out of bed in the middle of the night, but clearly it
would not be a total disaster.

No one is forced to use ARIN IRR, but it's worth asking the question:
why is ARIN a trustworthy steward of RPKI infrastructure if their IRR
is a serious liability to The Internet because of a simple issue like
not supporting password or PGP authentication?  Is this the reason
ARIN is spending time consulting their lawyers?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 1:09 PM, John Curran  wrote:
>  Please suggest your preferred means of IRR authentication to the ARIN
>  suggestion process: 
>  Alternatively, point to a best practice document from the operator
>  community for what should be done here. ARIN's work plan is very much
>  driven by community input, so that's what is needed here.

John,

I appreciate you taking time to respond to this while on vacation.
However, I think we all know that your response is not a "here is how
you tell us what to do," it's a "here is our cop-out response to make
an incredibly simple fix either never happen, or take six months to
make it through the ARIN process."

If you truly do not understand the posts regarding this matter, I will
summarize them for you very simply:
1) ARIN IRR is a tool that has operational impact; service providers
use it to build prefix-lists automatically, and if the data that
underlies those prefix-lists is corrupted, networks that use the ARIN
IRR will see their transit providers stop accepting their BGP
announcements overnight.  This is not a "some database might be
inaccurate but it's okay," problem; it is an operational problem.
Some peoples' networks depend on that data not becoming corrupted.
Specifically, every network that uses ARIN IRR.

2) ARIN IRR has effectively no security for record updates or deletes.
 Anyone who knows how to forge an email From: header can corrupt or
delete part or all of the ARIN IRR database at any time.  ARIN IRR is
the only database that I am aware of without support for at least
password authentication.  The standard toolset supports passwords
trivially.

3) If not supporting passwords was a business-driven decision, it was
a bad one, but perhaps a mistake born out of ignorance.  If it was a
technically-driven decision by the staff members responsible for
implementing and maintaining the ARIN IRR, those staff members are not
qualified to handle anything of an operational nature, and you would
be well-advised to find jobs for them that don't require any
attentiveness to operational security.

4) The "ARIN process" will almost certainly not be the route taken
when a change eventually arises.  Some black hat will eventually
decide it would be a clever prank to erase or corrupt the entire
database, and you will then be faced with three choices; a) implement
passwords immediately and not allow any updates from users who haven't
selected one; b) make the ARIN IRR read-only and effectively make it
useless; c) ignore the problem, at which point no ISPs will be willing
to mirror the ARIN IRR anymore, because its data is a liability, not
an asset.

I appreciate that there is a process to go through for proposing ARIN
policy changes, etc.  Your suggestion that this be used when
addressing an operational security matter is foolish and provides
plenty of ammo for people who say ARIN is ineffective (or worse.)

I suggest you take a moment to think about what the news coverage
might be if this eventually blows up in a big enough way to interest
news people.  If a bunch of ISPs go down overnight due to an ARIN
oversight, will some savvy reporter ask himself who at ARIN knew they
were running an operationally-important service with no security
mechanism at all?  Will he have much trouble finding out about a
mailing list discussion in which the CEO of ARIN glazed over the issue
and referred a whistle-blowing person to the ARIN policy process?
Will he then ask if ARIN is an effective steward of RPKI?  Will his
article assign blame to you personally?  Will he draw some link to
"Chinese interception of 15% of the Internet?"

Who knows how mainstream press would interpret such an event, if it
was big enough to attract attention.  If I were you, though, I would
not want my signature at the bottom of an email essentially telling
someone to go post on the correct mailing list.

I suggest you don't be the ARIN CEO that gets mud in his eye because
he didn't understand the value of a password over mail-from.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB? (IRR support & direction at ARIN)

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 6:27 PM, Randy Bush  wrote:
>>   Do you: 1) want IRR services, and if so, with what features?
>>           2) believe IRR services should be provided by ARIN?
>
> the irr is slightly useful today.  so, iff it is cheap and easy, arin
> providing an open and free instance is a public good.  again, iff it is
> easy and cheap.  and please do not waste time trying to 'fix' the irr,
> sad to say it's trying to make a silk purse out of a sow's ear.

I'm not suggesting that ARIN undertake a large and complex effort to
solve a bunch of issues with IRR.  All I am suggesting is that they
prevent anonymous bad guys with no inside information, special access,
or knowledge of passwords, from corrupting the data which some
networks choose to publish in ARIN IRR.

I am simply suggesting it is dangerous and irresponsible to run an IRR
with only MAIL-FROM authentication, and quite easy to also support
CRYPT-PW.  ARIN should either support passwords or immediately make
their IRR read-only and stop offering it as a service.  Imagine if
there was a Slashdot article or something about this, how long would
it take for some 14-year-old to erase the whole database, and how that
would pretty much force ARIN to make a choice anyway, but also, create
a lot of negative fall-out that might jeopardize trust in ARIN with
regard to other operational matters, like RPKI.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB? (IRR support & direction at ARIN)

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 6:48 PM, Randy Bush  wrote:
> jeff, i do not disagree that running an irr instance with only mail-from
> is s 1980s.  and, as mans points out, there is free software out
> there to do it (i recommend irrd).  but i do not see good cause for arin
> to spend anything non-trivial to fix a problem in an irr instance which
> is not used very much.  i.e. better to drop it than to spend non-trivial
> money to modernize it.

I agree that if ARIN thinks it would be "too costly" to support
password authentication, they should make the database read-only so
users will migrate away from it and no damage can be done by "bad
guys."

> but more to the point, by 'fix' it, i did not mean modernizing the auth
> method set.  i meant the content, syntax and semantics.

I understood what you meant, and again, I agree with you; there is no
reason to invest "a lot" of time and resources in something that
should be made obsolete by other work already in progress.  The "fix"
I want is simply eliminating the large liability by continuing to
allow updates with MAIL-FROM authentication.

I believe ARIN IRR actually does support MD5 authentication, but if
you email the ARIN IRR person, or go to ARIN's web site, you are told
that only MAIL-FROM is allowed.  So they probably already have the
appropriate technical mechanism in place AND JUST AREN'T USING IT, and
are actively discouraging users from utilizing it.  This would be an
example of ARIN's ineffectiveness when it comes to operational
matters, and is why I have real fear that RPKI may one-day be a
disaster because ARIN is an ineffective steward.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 7:33 PM, John Curran  wrote:
> My reason for responding is simply to make sure that ARIN is doing
> what the community wants.  I won't deny that this may take some time
> depending on exactly what is involved, but in my mind that is far
> better than not fixing the situation.

How will ARIN respond to operational security matters with regard to
RPKI infrastructure in the future?

What experience does ARIN have with operational security in the past?
When faced with DNS server vulnerabilities, did ARIN solicit community
feedback before patching the servers responsible for IN-ADDR.ARPA
zones administered by ARIN?  Or did ARIN treat this matter as a
legitimate, operational security concern, and apply whatever technical
solution was available and generally accepted by other organizations
administering DNS servers?

Why should an operational security issue with the ARIN IRR be handled
as a policy issue?

Do you know that I have emailed ARIN about this both recently and in
years past?  Am I the only person who has ever tried to bring this to
ARIN's attention?  I doubt that.

Are the personnel managing the ARIN IRR oblivious to the fact that
every other IRR database except ARIN supports at least some form of
password authentication?  Are these personnel qualified to handle
services with operational impact?

Do you, or they, know that ARIN's IRR technical infrastructure
actually does support password security, and that records exist in the
ARIN IRR database with MD5 authentication, but that email to ARIN
about this are answered with replies that only MAIL-FROM is possible?
Why does the ARIN web site make no mention of anything besides
MAIL-FROM?

> Thanks; I'm aware of the ARIN IRR and how operators in the community
> make use of it, and have run ISPs which have made use of the data
> for route filtering.

When you ran ISPs that made use of IRR data for route filtering, did
you use any kind of authentication when publishing and maintaining
your own records, or advise customers to use such?  Did the
possibility of malicious data corruption or erasure ever enter your
mind?

> Agreed; dropping me an email is a fine process for operational
> security matters.  Consider this one so reported.

What will the process be for handling operational security issues
regarding future RPKI infrastructure?  It is conceivable that there
may be no alternative to ARIN, in the ARIN region, for trusted routing
information data in the future.  Today, we can choose not to use ARIN
IRR, and the huge majority of networks who publish IRR data use their
ISP databases or MERIT RADB.  Are we faced with the possibility that
ARIN simply doesn't have personnel capable of handling operational
services, yet are forcing ARIN down a road that may make them a sole
source of something we all need?  If so, perhaps this is a very bad
idea in need of further debate.

I think the mentality at ARIN is one of paper-pushers and policy guys.
 That's perfectly fine for an organization whose main function is ...
processing paperwork and allocating IP addresses.  It is perhaps a
very bad idea to ask ARIN to do operational things which they are very
clearly unprepared to handle, to such an extent that they may need
additional or different personnel, and really need to change their
mentality.

I understand that the technical side of the RPKI implementation at
ARIN is most likely entrusted to Paul Vixie and ISC, which is a good
thing.  I never read an email from Paul saying, "I think we need to
solicit feedback before we patch this BIND issue."  DNSSEC progress
has taken a very long time, but that hasn't stopped ISC from
continuing to provide quick technical solutions to immediate technical
problems.  What really worries me is ... if there is some serious
issue with RPKI infrastructure in the future, will ARIN be able to
solve it in an operational time-frame, or won't they?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB?

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 10:47 PM, John Curran  wrote:
> Jeff - ARIN does indeed have folks who worry about whether the policy
> development process is being followed.  We also have folks who actually
> implement the policy and issue number resources.

And we all agree that this is ARIN's primary role, and what ARIN,
organizationally, has been built to be good at.  This is what members
consider when electing the BoT and no doubt drives ARIN's day-to-day
business and technical decisions.

> is that we also have quite a few folks who have run production operational
> services both for the Internet and other mission-critical environments.

What does ARIN, as an organization, do that has short-term operational
impact on its members?  Two things that I am aware of: IN-ADDR.ARPA
delegation and IRR.  One of these things gives people no reason to
complain.  The other is demonstrably insecure in a manner that could
have really serious, and embarrassing, consequences, both financial
for the members, and in terms of peoples' confidence in ARIN.

> I'm not surprised that the IRR allows plaintext passwords, but am myself
> stunned if indeed we require them, since that disallows even a modicum of
> protection from trivial acts of sabotage.  Rather than repeat what lack
> of information there is on the web site in regards to what forms of IRR
> authentication is available, I will go determinate the state of reality
> and post back here asap. At a minimum, we need much clearer documentation,
> but if more is required, we'll get it fixed asap.

Thanks, I am glad you are now looking into this.  To be clear, it's
not just "plain text passwords."  There aren't any passwords for the
majority of objects.  The ARIN documentation indicates that only
MAIL-FROM is supported.  When asked about this, ARIN personnel who
respond to rt...@arin.net reply that yes, MAIL-FROM is the only
authentication mechanism supported, and that no, there is no support
for passwords (good) or PGP (also good, but too complicated for some
users.)

This isn't simply an issue of "plain text passwords."  Your mechanism
is MAIL-FROM, which means the only check that is done on
update/add/delete requests is the From: header.  The ARIN database,
which is publicly mirrored, contains the email addresses that must be
used to add/update/delete objects maintained by a given mntner:
object.  All you have to do to corrupt or erase a record is look up
the record you want to corrupt in the IRR, then look up that mntner,
then forge an email from the auth: MAIL-FROM listed in that mntner
record.  It's dead simple and it is not "plain text passwords," it is
no passwords at all.

The reason I am still posting is I am deeply concerned about the lack
of technical and management competence needed to let this happen in
the first place.  You shouldn't seriously believe that no ARIN staffer
ever thought about this, while also believing that ARIN is currently
capable of administering RPKI, by its very nature and as its primary
goal, to improve operational network security.

For this reason, I think your true task is not simply to address the
IRR issue, but to change the mentality at ARIN.  If you do have
technically skilled personnel, something is preventing them from being
effective.  If there isn't a management or cultural problem stopping
folks from speaking up, then, quite frankly, I think you may be
greatly over-estimating the technical savvy of ARIN staff.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB? (IRR support & direction at ARIN)

[Jeff Wheeler]

On Sun, Jan 9, 2011 at 11:00 PM, Charles N Wyble
 wrote:
> So why hasn't this happened already? If it's so easy, then all the
> normal actors that like to cause us late nights would have struck already.

As most of us in the net ops community know, there are many
vulnerabilities that are very much non-obvious to a black hat guy used
to DDoSing with botnets or exploiting the latest common daemon
vulnerability.  That is no assurance that this vulnerability will
never be exploited.  The very fact that we are talking about it on
this mailing list (unfortunately) raises the chances that it will
happen.  If there was an article on Slashdot, I bet significant
corruption pranks or deliberate, malicious erasure would happen inside
of a week.  If I spent 15 minutes making a "HOWTO anonymously delete
an ISP from the ARIN IRR with a telnet client and an open proxy" and
spread it around to some IRC bad-guys, you can be assured we would be
talking about damage control, not prevention, by tomorrow.  Finally,
anyone who has ever 1) learned how email works; and 2) learned how to
update their own IRR objects via email; can do it without reading
anything, and has probably realized this vulnerability on their own
years ago.

> So I don't think ARIN should spend it's limited resources on anything to
> do with it's copy of the IRR. In fact I'm not sure why they even operate
> one. It seems to be the realm of service providers to do so.

It is desirable to publish your IRR records in a neutral database, as
opposed to a service provider database.  Let's say I am a Level3
customer and I use their IRR.  A year goes by, and I don't renew my
contract with Level3, I instead start buying transit from AT&T.  Well,
AT&T does not operate an IRR database.  Now I have to find a new place
to publish my IRR data, *and* my new transit provider doesn't offer it
as a service.  If I have a need for IRR, I had better hope one of my
other transit providers offers me a database, or use RADB, ALTDB, or
another third-party database.

This is why MERIT has a bunch of customers paying annual fees for
RADB, a valuable service; and why some great folks volunteer their
time to maintain the ALTDB.  It is also no doubt the reason ARIN has
an IRR database, but unfortunately, the ARIN IRR is a liability, not
an asset, to the net ops community.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: AltDB? (IRR support & direction at ARIN)

[Jeff Wheeler]

On Mon, Jan 10, 2011 at 12:37 PM, Jon Lewis  wrote:
> On Sun, 9 Jan 2011, Charles N Wyble wrote:
>
>>> I am simply suggesting it is dangerous and irresponsible to run an IRR
>>> with only MAIL-FROM authentication, and quite easy to also support
>>> CRYPT-PW.  ARIN should either support passwords or immediately make
>
> The trouble is, since the DES crypt passwords are publicly accessible, even
> CRYPT-PW is not much security.  I suspect with a copy of the db, a passsword
> cracking program, and some modest computing capacity, you could crack all

DES crypt() is not completely trivial yet, but I agree, it is far from
state-of-the-art.  It is substantially superior to MAIL-FROM.  In
addition, MERIT reduced this problem by simply filtering out the
hashes from the RADB.db file and whois output (and presumably also,
the www.radb.net tools.)

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefix lengths

[Jeff Wheeler]

Richard's employer is exactly the kind of organization that has not
been able to effectively multi-home their discrete branch-offices on
the IPv4 Internet, because RIR allocation policy set the bar for
receiving IPv4 addresses for those small locations just high enough to
steer us away from that "feature" or "problem," depending on how you
look at it.

If every Richard Barnes announces a few dozen /48s into the global BGP
table, it will not be long before the 300k+ IPv4 BGP table looks neat
and organized, and the CIDR Report will come out each week with a
message begging e.g. Starbucks to aggregate their coffee shop wireless
hot-spots, instead of shaming Bell South for having a large number of
de-aggregates for their substantial ISP business.

Most people do not know about the "multi-homing feature" designed into
IPv6.  Most people who do, seem to agree that it may not see enough
practical use to have meaningful impact on routing table growth, which
will no longer be kept in check by a limited pool of IP addresses and
policies that make it a little difficult for a very small network to
become multi-homed.

This may be another looming IPv6 headache without a sufficient
solution to set good practices now, before deployment sky-rockets.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: ARIN IRR Authentication (was: Re: AltDB?)

[Jeff Wheeler]

On Thu, Jan 27, 2011 at 10:00 PM, John Curran  wrote:
> Based on the ARIN's IRR authentication thread a couple of weeks ago, there
> were suggestions placed into ARIN's ACSP process for changes to ARIN's IRR
> system. ARIN has looked at the integration issues involved and has scheduled
> an upgrade to the IRR system that will accept PGP and CRYPT-PW authentication
> as well as implementing notification support for both the mnt-nfy and notify
> fields by the end of August 2011.

I'm glad to see that a decision was made to improve the ARIN IRR,
rather than stick to status-quo or abandon it.  However, this response
is essentially what most folks I spoke with off-list imagined: You
have an immediate operational security problem which could cause
service impact to ARIN members and others relying on the ARIN IRR
database, and fixing it by allowing passwords or PGP to be used is not
very hard.

As I have stated on this list, I believe ARIN is not organizationally
capable of handling operational issues.  This should make everyone
very worried about any ARIN involvement in RPKI, or anything else that
could possibly have short-term operational impact on networks.  Your
plan to fix the very simple IRR problem within eight months is a very
clear demonstration that I am correct.

How did you arrive at the eight month time-frame to complete this project?

Can you provide more detail on what CRYPT-PW hash algorithm(s) will be
supported?  Specifically, the traditional DES crypt(3) is functionally
obsolete, and its entire key-space can be brute-forced within a few
days on one modern desktop PC.  Will you follow the practice
established by several other IRR databases (including MERIT RADB) and
avoid exposing the hashes by way of whois output and IRR database
dumps?

If PGP is causing your delay, why don't you address the urgent problem
of supporting no authentication mechanism at all first, and allow
CRYPT-PW (perhaps with a useful hash algorithm) and then spend the
remaining 7.9 months on PGP?

The plan and schedule you have announced is indefensible for an
operational security issue.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Level 3's IRR Database

[Jeff Wheeler]

On Sun, Jan 30, 2011 at 3:23 AM, Andrew Alston  wrote:
> I've just noticed that Level 3 is allowing people to register space in its 
> IRR database that A.) is not assigned to the people registering it and B.) is 
> not assigned via/to Level 3.

This is not unique to Level3 -- it is the industry standard practice
and has been since the dawn of time.  You must be a Level3 customer to
have a mntner: for publishing to their IRR database (in theory.)

Since there isn't an automatic mechanism for verifying that a given
ISP is really allowed to originate a route (or provide transit for an
AS, etc.) there is simply no practical way to change this at this
time, without processing updates manually (and introducing human error
into that yes/no authorization check.)

IRR is a convenience that many networks rely on.  When done correctly,
this is not a bad idea by any means.

In theory, RPKI will fix the real problem you are addressing -- that
it is really difficult to verify whether or not a neighboring AS is
allowed to carry a given route.  In practice, vendors need to support
it on routers, networks need to upgrade, ARIN (and other RIRs) need to
do their part, and thousands of auto-pilot networks will need to be
hand-held by their ISPs in order to make this happen.  How soon theory
can become reality is not easy to predict.  How many networks have
ubiquitous support for 32 bit ASN?  IPv6?  RPKI is a bastard thing
created out of a perceived (perhaps correctly) need for real security,
when in fact basically all of the events that have led to its creation
(except for some scare-tactic papers and presentations) were not
deliberate.

This brings me to my point, which is that IRR is very good for
preventing accidents and automating some common tasks.  It should be
"secure" to a point, but just because a route: object exists does not
mean that mntner: really has authority over that address space.  You
can pretty much rely on the fact that the given origin AS is
intentionally announcing the route, as opposed to leaking it by
accident.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: [arin-announce] ARIN Resource Certification Update

[Jeff Wheeler]

On Sun, Jan 30, 2011 at 12:40 PM, Owen DeLong  wrote:
> Because they publish data you have signed. They don't have the ability
> to modify the data and then sign that modification as if they were you if
> they aren't holding the private key. If they are holding the private key,
> then, you have, in essence, given them power of attorney to administer
> your network.
>
> If you're OK with that, more power to you. It's not the trust model I would
> prefer.

I suspect that many users would prefer to trust ARIN with their
private keys, if offered that choice.  The reasons are simple;
adoption will be more wide-spread if RPKI is easier to do; and as we
all know, there are an awful lot of BGP networks which are:
* on auto-pilot, with no clued in-house staff and no stable
relationships with outside clue
* driven by people who are somewhere between totally clueless and
capable of understanding public-key encryption
* driven by over-worked people who simply don't have time for another
to-do of any complexity

Many users would benefit from the kind of hosted service that is made
available by, for example, RIPE.  In fact, if they felt they could
trust ARIN (or any alternative service which may exist), most of my
clients would be perfectly fine with such a service, and I would not
advise them to do otherwise without a valid business reason and a
belief that equal or superior security would be provided by not using
such a hosted service.  Since ARIN holds ultimate authority over the
ISP's address space anyway, if ARIN's private keys become compromised,
whether or not you held onto your own keys will not matter to the rest
of the world.

If I understand correctly, John has expressed that ARIN's concern is
they may be sued if their hosted service fails to perform, and that
ordinary contractual language may be unable to limit damages if the
reality is that the service-customer has no other choice but to use
the ARIN service.  This is clearly not a legitimate concern if there
is an alternative to such an ARIN hosted service, such as using no
hosted service at all, or the possibility of using another one.

I don't see how the lack of ARIN providing a hosted service
immediately in any way prevents them from doing so in the future.  If
widespread RPKI adoption doesn't happen and a few more accidental or
intentional YouTube black-holes do happen, it seems likely that
stakeholders will encourage ARIN to do more, and a hosted service
would be an obvious step to increase adoption.

As you know, my comfort level with ARIN handling tasks of an
operational nature is not high; but if they are going to participate
in RPKI in any way, I think they should be capable of performing
similarly to RIPE.  If not, we should be asking ourselves either 1)
why would anyone trust RIPE with their keys; or 2) why is RIPE more
trustworthy than ARIN?

If the answer to that is RIPE is significantly more competent than
ARIN (most folks I know are of this belief) then this discussion
should not be about one technical effort.  Instead, it should be about
how to make ARIN better.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Consequences of BGP Peering with Private Addresses

[Jeff Wheeler]

On Wed, Jun 15, 2011 at 12:47 PM, James Grace  wrote:
> So we're running out of peering space in our /24 and we were considering 
> using private /30's for new peerings.  Are there any horrific consequences to 
> picking up this practice?

I agree with other posters that this is not a good practice.  Is it
somehow not possible for you to obtain additional address space?  Can
you not use neighbor-assigned /30s more frequently to avoid exhausting
your existing allocation?

For eBGP neighbors, I would sooner use non-unique /30s than utilize
RFC1918 space.  While this would not allow for correct reverse DNS,
and traceroute would be less obvious, it has fewer disadvantages than
assigning RFC1918 for your peer link-nets.  You will need to re-write
next-hop towards iBGP neighbors, though (using next-hop-self or
translating to internal numbers for routing protocol use) and you
should not re-use the same /30 twice on the same ASBR.

This may sound crazy, and it is certainly not an ideal way of doing
things; but it is an alternative worth consideration as networks
exhaust their available IPv4.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: ICANN to allow commercial gTLDs

[Jeff Wheeler]

On Sat, Jun 18, 2011 at 12:04 AM, George B.  wrote:
> I think I will get .payme  and make sure coke.payme, pepsi.payme,
> comcast.payme, etc. all get registered at the low-low price of
> $10/year.  All I would need is 100,000 registrations to provide me
> with a million dollar a year income stream for the rest of my life.

I have read this thread, but certainly not any ICANN garbage.  It
seems to me that a TLD for a brand, like Coca-Cola, would not be used
in the same way as GTLDs.  Will George actually be allowed to carve up
his own TLD and sell bits of it to anyone who is willing to click a
checkbox on GoDaddy.com?  Obviously there is not any technical
limitation in place to prevent this, but will there be legal / "layer
9" limitations?

I kinda figured additional GTLDs is not very useful given that
probably every domain registrar drives customers to "protect their
brand," avoid phishing attacks against their customers, etc. by buying
not only example.com, but also net|org|biz|etc.  I imagine that
registrars may be really excited about this idea, because it
represents additional fees/revenue to them.  I can't understand why it
is good for anyone else.  Does McDonald's really want to print
http://mcdonalds/ or www.mcdonalds instead of www.mcdonalds.com on
their soft drink cups and TV ads?

Is Owen so disconnected from reality that he thinks the chain with the
golden arches is spelled "MacDonald's?"

I don't particularly care about the intellectual property questions
(in the context of NANOG) but if you really want to bang your head
against that, I suggest reading about the current trademark status of
"Standard Oil."  In short, it remains a legally protected mark but has
several distinct owners throughout the United States -- a result of
the break-up.  "Waffle House" is a little complex, too.  Somehow the
GTLD system continues to function.  I imagine the relevant authorities
are capable of figuring out who should be allowed to register which
brand-TLD.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Wacky Weekend: NERC to relax power grid frequency strictures

[Jeff Wheeler]

On Sun, Jun 26, 2011 at 12:23 AM, Alex Rubenstein  wrote:
> At least here in JCPL territory (northern NJ), closed transition is frowned 
> upon. Too much risk, they think. They are correct, really, but the risk is 
> mostly yours. If you lock to the utility out-of-phase, you will surely lose 
> and they will surely win. The fault you create that they will see will 
> probably not hurt them. Unless it is extraordinarily large and you are very 
> close to the nearest substation.

Utilities concern themselves with not only their gear and your gear,
but also your neighbor's gear.  I would not like to be next-door to a
large genset that is connected to the grid out-of-phase.  My equipment
would be affected by such an event.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Why is IPv6 broken?

[Jeff Wheeler]

On Sat, Jul 9, 2011 at 5:25 PM, Bob Network  wrote:
> Why is IPv6 broken?

You should have titled your thread, "my own personal rant about
Hurricane Electric's IPv6 strategy."  You may also have left out the
dodgy explanation of peering policies and technicalities, since these
issues have been remarkably static since about 1996.  The names of the
networks change, but the song remains the same.  This is not a novel
subject on this mailing list.  In fact, there have been a number of
threads discussing HE's practices lately.  If you are so interested in
them, I suggest you review the list archive.

There are quite a few serious, unresolved technical problems with IPv6
adoption besides a few networks playing chicken with their collective
customer-bases.  The lack of will on the part of vendors and operators
to participate in the IETF process, and make necessary and/or
beneficial changes to the IPv6 standards, has left us in a situation
where IPv6 implementation produces networks which are vulnerable to
trivial DoS attacks and network intrusions.

The lack of will on the part of access providers to insist on
functioning IPv6 support on CPE and BRAS platforms has even mid-sized
ISPs facing nine-figure (as in, hundred-million-dollars) expenses to
forklift-upgrade their access networks and end-user equipment, at a
time when IPv6 seems to be the only way to continue growing the
Internet.

The lack of will on the part of major transit networks, including
Savvis, to deploy IPv6 capabilities to their customers, means that
customers caught in multi-year contracts may have no option for native
connectivity.  Cogent's policy of requiring a new contract, and from
what I am still being told by some European customers, new money, from
customers in exchange for provisioning IPv6 on existing circuits,
means a simple technical project gets caught up in the complexities of
budgeting and contract execution.

If you believe that the most serious problem facing IPv6 adoption is
that HE / Level3 / Cogent don't carry a full table, you are living in
a fantasy world.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Sun, Jul 10, 2011 at 3:45 PM, Owen DeLong  wrote:
> Number two: While anyone can participate, approaching IETF as an
> operator requires a rather thick skin, or, at least it did the last couple
> of times I attempted to participate. I've watched a few times where

I am subscribed to the IDR (BGP, etc.) and LISP lists.  These are
populated with different people and cover entirely different topics.
My opinion is the following:

* The IDR list is welcoming of operators, but whether or not your
opinion is listened to or included in the process, I do not know.
Randy Bush, alone, posts more on this list than the sum of all
operators who post in the time I've been reading.  I think Randy's
influence is 100% negative, and it concerns me deeply that one
individual has the potential to do so much damage to essential
protocols like BGP.  Also, the priorities of this list are pretty
fucked.  Inaction within this working group is the reason we still
don't have expanded BGP communities for 32 bit ASNs.  The reason for
this is operators aren't participating.  The people on the list or the
current participants of the WG should not be blamed.  My gripe about
Randy Bush having the potential to do huge damage would not exist if
there were enough people on the list who understand what they're doing
to offer counter-arguments.

> operators were shouted down by purists and religion over basic
> real-world operational concerns. It seems to be a relatively routine
> practice and does not lead to operators wanting to come back to
> an environment where they feel unwelcome.

I have found my input on the LISP list completely ignored because, as
you suggest, my concerns are real-world and don't have any impact on
someone's pet project.  LISP as it stands today can never work on the
Internet, and regardless of the fine reputations of the people at
Cisco and other organizations who are working on it, they are either
furthering it only because they would rather work on a pet project
than something useful to customers, or because they truly cannot
understand its deep, insurmountable design flaws at Internet-scale.
You would generally hope that someone saying, "LISP can't work at
Internet-scale because anyone will be able to trivially DoS any LISP
ITR ('router' for simplicity), but here is a way you can improve it,"
well, that remark, input, and person should be taken quite seriously,
their input examined, and other assumptions about the way LISP is
supposed to work ought to be questioned.  None of this has happened.
LISP is a pet project to get some people their Ph.D.s and keep some
old guard vendor folks from jumping ship to another company.  It is a
shame that the IETF is manipulated to legitimize that kind of thing.

Then again, I could be wrong.  Randy Bush could be a genius and LISP
could revolutionize mobility.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Why is IPv6 broken?

[Jeff Wheeler]

On Mon, Jul 11, 2011 at 3:25 AM, Tom Hill  wrote:
> On Sun, 2011-07-10 at 10:14 -0400, Jeff Wheeler wrote:
>> Cogent's policy of requiring a new contract, and from what I am still
>> being told by some European customers, new money, from customers in
>> exchange for provisioning IPv6 on existing circuits, means a simple
>> technical project gets caught up in the complexities of budgeting and
>> contract execution.
>
> "Can we have IPv6 transit?"
> "Yes, please turn up a session to.."
>
> That was asking Cogent for IPv6 dual-stack on our existing IPv4
> transit.

I continue to hear different.  In my first-hand experience just about
three weeks ago, I was told by Cogent that I need to execute a new
contract to get IPv6 added to an existing IPv4 circuit (U.S.
customer.)  This turned a simple pilot project with only a few I.T.
folks involved into, well, I'm still waiting on this new contract to
be executed.  I'm not surprised.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Mon, Jul 11, 2011 at 3:18 PM, William Herrin  wrote:
> On the other hand, calling out ops issues in RFCs is a modest reform
> that at worst shouldn't hurt anything. That beats my next best idea:

I think if this were done, some guy like me would spend endless hours
arguing with others about what should and should not be documented in
this proposed section, without it actually benefiting the process or
the improving the underlying protocol function / specification.  Let
me give you an example:

BGP Messages, which are up to 4KB, need to be expanded to support
future features like as-path signing.  Randy Bush proposes to extend
them to 65,535 octets, the maximum size without significantly changing
the message header.  This raises a few concerns which I label as
operational, for example, off-by-one bugs in code can fail to be
detected by a neighboring BGP speaker in some circumstances, because
an age-old (since BGP 1) idiot check in the protocol is being silently
removed.

If you ask me, that is operational and belongs in such a section.  I'm
sure others will disagree.  So we would have a bunch of arguing over
whether or not to call this out specifically.

Another person believes that expanding the message will affect some
vendors' custom TCP stacks, due to window size considerations.  I
might think that is a developer problem and the affected vendors
should fix their crappy TCP implementations, but it might produce
unusual stalling problems, etc. which operators have to troubleshoot.
Is that an operational issue?  Should it be documented?

There can be many "operational concerns" when creating or modifying a
protocol specification, and every person won't agree on what belongs
and what doesn't.  However, I do not think the requirement to document
them will improve the process or the protocols.  It will only add
work.

Besides, you want "IETF people" who are claimed not to understand
operational problems to figure them out and document them in the RFCs?
 I do not think this will be helpful.  More hands-on operators
participating in their process is what is needed.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Mon, Jul 11, 2011 at 3:35 PM, Leo Bicknell  wrote:
> The IETF does not want operators in many steps of the process.  If
> you try to bring up operational concerns in early protocol development
> for example you'll often get a "we'll look at that later" response,
> which in many cases is right.  Sometimes you just have to play with
> something before you worry about the operational details.  It also

I really don't understand why that is right / good.  People get
personally invested in their project / spec, and not only that, vendor
people get their company's time and money invested in
proof-of-concept.  The longer something goes on with what may be
serious design flaws, the harder it is to get them fixed, simply
because of momentum.

Wouldn't it be nice if we could change the way that next-header works
in IPv6 now?  Or get rid of SLAAC and erase the RFCs recommending /80
and /64 from history?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Mon, Jul 11, 2011 at 5:12 PM, Owen DeLong  wrote:
> No... I like SLAAC and find it useful in a number of places. What's wrong
> with /64? Yes, we need better DOS protection in switches and routers

See my slides http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf for
why no vendor's implementation is effective "DOS protection" today and
how much complexity is involved in doing it correctly, which requires
not only knobs on routers, but also on layer-2 access switches, which
is not easy to implement.  It's a whole lot smarter to just configure
a smaller network when that is practical.  In fact, that advice should
be "the standard."

I really don't understand why we need SLAAC.  I believe it is a relic
of a mindset when a DHCP client might have been hard to implement
cost-effectively in a really light-weight client device (coffee pot?
wrist-watch?)  Or when running a DHCP server was some big undertaking
that couldn't be made not only obvious, but transparent, to SOHO users
buying any $99 CPE.

I do understand why SLAAC needs /64.  Okay, so configure /64 on those
networks where SLAAC is utilized.  Otherwise, do something else.
Pretty simple!  Again, please see my slides.

> to accommodate some of the realities of those decisions, but, that's not
> to say that SLAAC or /64s are bad. They're fine ideas with proper
> protections.

The proper protections are kinda hard to do if you have relatively
dumb layer-2 access switches.  It is a lot harder than RA Guard, and
we aren't ever likely to see that feature on a large base of installed
"legacy" switches, like Cisco 2950.  Replacing those will be
expensive.  We can't replace them yet anyway because similar switches
(price) today still do not have RA Guard, let alone any knobs to
defend against neighbor table churn, etc.  I'm not sure if they ever
will have the later.

> I'm not sure about the /80 reference as I haven't encountered that
> recommendation outside of some perverse ideas about point-to-point
> links.

This is because you didn't follow IPv6 progress until somewhat
recently, and you are not aware that the original suggestion for
prefix length was 80 bits, leaving just 48 bits for the host portion
of the address.  This was later revised.  It helps to know a bit of
the history that got us to where we are now.

It was originally hoped, by some, that we may not even need NDP
because the layer-2 adjacency would always be encoded in the end of
the layer-3 address.  Some people still think vendors may get us to
that point with configuration knobs.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Mon, Jul 11, 2011 at 7:48 PM, Jimmy Hess  wrote:
> If every vendor's implementation is vulnerable to a NDP Exhaustion
> vulnerability,
> how come the behavior of specific routers has not been documented 
> specifically?

Well, I am in the business of knowing the behavior of kit being
considered by my clients for their applications.  Every box breaks
when tested, period.  I imagine you have tested zero, thus you have no
data of your own to go on.  No vendors are rushing to spend money on
"independent" testing laboratories to produce reports about this,
because they pretty much all know their boxes will break (or are not
even aware of the potential problem, in the case of a few scary
vendors.)

> If  "zero" devices are not vulnerable, you came to this conclusion
> because you tested
> every single implementation against IPv6 NDP DoS,  or?

Although I have tested many routers to verify my thinking, if you
actually read the slides and understand how routers work, you too will
know that every router is vulnerable.  If you don't know, you don't
understand how routers work.  It's that simple.

> How come there are no security advisories.
> What's the CWE or CVE number for this vulnerability?

Again, no one is interested in this problem yet because vendors really
don't want their customers to demand more knobs.  Cisco is the only
vendor who has done anything at all.  If you read about their knob,
you immediately realize that it is a knob to control the failure mode
of the box, not to "fix" anything.  Why?  It can't be "fixed" without
not using /64 (or similar) or going to the extreme lengths I outline
in those slides.

> It would be useful to at least have the risk properly described, in
> terms of what
> kind of DoS condition could arise on specific implementations.

Let's take 6500/SUP720 for example.  On this platform, a policer is
shared between the need to resolve ARP entries and ND table entries.
If you attack a dual-stack SUP720 box it will break not only IPv6
neighbor resolution, but also IPv4 neighbor resolution.  This is
pretty much the "worst-case scenario" because not only will your IPv6
break, which may annoy customers but not be a disaster; it will also
break mission-critical IPv4.  That's bad.  Routing-protocol
adjacencies can be affected, disabling not just some hosts downstream
of the box, but also its upstream connectivity.  It doesn't get any
worse than that.

You are right to question my statements.  I'm not an independent lab
doing professional tests and showing the environment and conditions of
how you can reproduce the results.  I'm just a guy helping my clients
decide what kit to buy, and how they should configure their networks.
The only reason I have bothered to produce slides is because we are at
a point where we have end-customers questioning our reluctance to
provision /64 networks for mixed-use data-center LANs, and until
vendors actually do something to address this, or "the standard"
changes, I need to increase awareness of this problem so I am not
forced to deploy a broken design on my own networks the way a lot of
other clueless people are.

Again, this is only hard to understand (or accept) if you don't know
how your routers work.
* why do you think there is an ARP and ND table?
* why do you think there are policers to protect the CPU from
excessive ARP/ND punts or traffic?
* do you even know the limit of your boxes' ARP / ND tables?  Do you
realize that limit is a tiny fraction of one /64?
* do you understand what happens when your ARP/ND policers are reached?
* did you think about the impact on neighboring routers and protocol
next-hops, not just servers?
* did you every try to deploy a /16 on a flat LAN with a lot of hosts
and see what happens?  Doesn't work too well.  A v6 /64 is 281
trillion times bigger than a v4 /16.  There's no big leap of logic
here as to why one rogue machine could break your LAN.

There is no router which is not vulnerable to this.  If you don't
believe me, read the Cisco documentation on their knob limiting ND
entries per interface, after which there may be service impact on that
interface.  That's the best anyone is doing right now.  Of course,
vendors understand that we, as customers, can configure a subnet
smaller than /64.  They are leaving us open to link-local issues right
now even with a smaller global subnet size, but at least that cannot
be exploited from "the Internet."  And as it happens, exactly the same
features / knobs are needed to "fix" both problems with /64, and with
link-local neighbor learning.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Tue, Jul 12, 2011 at 11:42 AM, Leo Bicknell  wrote:
> I'll pick on LISP as an example, since many operators are at least
> aware of it.  Some operators have said we need a locator and identifier
> split.  Interesting feedback.  The IETF has gone off and started
> playing in the sandbox, trying to figure out how to make that go.

As an operator (who understands how most things work in very great
detail), I found the LISP folks very much uninterested in my concerns
about if LISP can ever be made to scale up to "Internet-scale," with
respect to a specific DDoS vector.  I also think that an explosion of
small, multi-homed SOHO networks would be a disaster, because we might
have 3 million FIB instead of 360k FIB after a few years.  These
things are directly related to each-other, too.

So I emailed some LISP gurus off-list and discussed my concern.  I was
encouraged to post to the LISP IETF list, which I did.  To my great
surprise, not one single person was interested in my problem.  If you
think it is a small problem, well, you should try going back to
late-1990s flow-cache routing in your data-center networks and see
what happens when you get DDoS.  I am sure most of us remember some of
those painful experiences.

Now there is a LISP "threats" draft which the working group mandates
they produce, discussing various security problems.  The current paper
is a laundry list of "what if" scenarios, like, what if a malicious
person could fill the LISP control-plane with garbage.  BGP has the
same issue, if some bad guy had enable on a big enough network that
their peers/transits don't filter their routes, they could do a lot of
damage before they were stopped.  This sometimes happens even by
accident, for example, some poor guy accidentally announcing 12/9 and
giving AT&T a really bad day.

What it doesn't contain is anything relevant to the special-case DDoS
that all LISP sites would be vulnerable to, due to the IMO bad
flow-cache management system that is specified.  I am having a very
great deal of trouble getting the authors of the "threats" document to
even understand what the problem is, because as one of them put it, he
is "just a researcher."  I am sure he and his colleagues are very
smart guys, but they clearly do not remember our 1990s pains.

That is the "not an operator" problem.  It is understandable.

Others who have been around long enough simply dismiss this problem,
because they believe the unparalleled benefits of LISP for mobility
and multi-homing SOHO sites must greatly out-weigh the fact that,
well, if you are a content provider and you receive a DDoS, your site
will be down and there isn't a damn thing you can do about it, other
than spec routers that have way, way more FIB than the number of
possible routes, again due to the bad caching scheme.

The above is what I think is the "ego-invested" problem, where certain
pretty smart, well-intentioned people have a lot of time, and
professional credibility, invested in making LISP work.  I'm sure it
isn't pleasing for these guys to defend their project against my
argument that it may never be able to reach Internet-scale, and that
they have missed what I claim is a show-stopping problem with an easy
way to improve it through several years of development.  Especially
since I am a guy who did not ever participate in the IETF before,
someone they don't know from a random guy on the street.

I am glad that this NANOG discussion has got some of these LISP folks
to pay more attention to my argument, and my suggested improvement (I
am not only bashing their project; I have positive input, too.)
Simply posting to their mailing list once and emailing a few draft
authors did not cause any movement at all.  Evidently it does get
attention, though, to jump up and down on a different list.  Go
figure!

If operators don't provide input and *perspective* to things like
LISP, we will end up with bad results.

How many of us are amazed that we still do not have 32:32 bits BGP
communities to go along with 32 bit ASNs, for signalling requests to
transit providers without collision with other networks' community
schemes?  It is a pretty stupid situation, and yet here we are, with
32 bit ASN for years, and if you want to do advertisement control with
32 bit ASNs used, you are either mapping your 32 bit neighbors to
special numbers, or your community scheme can overlap with others.

That BGP community problem is pretty tiny compared to, what if people
really started rolling out something new and clever like LISP, but in
a half-baked, broken way that takes us back to 1990s era of small DDoS
taking out whole data-center aggregation router.  A lot of us think
IPv6 is over-baked and broken, and probably this is why it has taken
such a very long time to get anywhere with it.  But ultimately, it is
our fault for not participating.  I am reversing my own behavior and
providing input to some WGs I care about, in what time I have to do
so.  More operators should do the same.  

Re: in defense of lisp (was: Anybody can participate in the IETF)

[Jeff Wheeler]

On Wed, Jul 13, 2011 at 2:27 AM, Randy Bush  wrote:
>> I fear that at its worst and most successful, LISP ensures ipv4 is the
>> backbone transport media to the detriment of ipv6 and at its best, it
>> is a distraction for folks that need to be making ipv6 work, for real.
>
> i suspect that a number of lisp proponents are of that mind.  i do not
> think it does a service to the internet.

My understanding is that transport over v6 is indeed on everyone's
mind and absolutely is a goal for all the LISP people.  So on this
particular point, your concern is being addressed.

What LISP has not done is actually improve the root problem of scaling
up the number of multi-homed networks or locators.  The cache scheme
works if you imagine an ideal Internet where there is no DoS, but
otherwise, it does not work.  All the same problems of flow-cache
routing still exist and LISP actually makes them worse in some cases,
not better.  It also adds huge complexity and risk but what value it
adds (outside of VPN-over-Internet) is questionable at best.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

Luigi, you have mis-understood quite a bit of the content of my
message.  I'm not sure if this is of any further interest to NANOG
readers, but as it is basically what seems to go on a lot, from my
observations of IETF list activity, I'll copy my reply to the list as
you have done.

On Wed, Jul 13, 2011 at 4:08 AM, Luigi Iannone
 wrote:
> Granted. You are the real world expert. Now can you stop repeating this in
> each email and move on?

No.  This is a point that needs to be not only made, but driven home.
You do not understand how routers work, which is why you are having
such difficulty understanding the severity of this problem.  The
lisp-threats work you have done is basically all control-plane /
signalling issues, and no data-plane issues.  This is not a
coincidence; it is because your knowledge of the control-plane side is
good and of the data-plane is weak.

> This is completely false. Several people gave credit to you about the
> existence of the threat you pointed out.

Really?  In April, when I posted a serious problem, and received no
replies?  Now, the original folks who I discussed this with, before
ever posting to the IETF LISP list, are finally seeking clarification,
because apparently there may have been some confusion in April,
possibly leading to their total dismissal of this as a practical
concern.

> This is again false. We had mail exchange both privately and on the
> mailinglist. We proposed to you text to be added to the threats draft but
> you did not like it. We are asking to propose text but we have no answer
> from you on this point.

Actually, you classified this as an implementation concern, which is
false.  You have said yourself that this is why you believe it
deserves just one sentence, if that, in the lisp-threats draft.  This
is not an implementation-specific concern, it is a design flaw in the
MS negative response scheme, which emerges to produce a trivial DoS
threat if LISP ever scales up.

>> Now there is a LISP "threats" draft which the working group mandates
>> they produce, discussing various security problems.  The current paper
>> is a laundry list of "what if" scenarios, like, what if a malicious
>> person could fill the LISP control-plane with garbage.  BGP has the
>
> So you are saying that BGP can be victim of similar attacks/problem
> still... if you are reading this email it means that the Internet is still
> running...

This is where I believe you are mis-reading my message.  Your threats
draft covers legitimate concerns which also exist in the current
system that is widely deployed, which is largely, BGP plus big FIB.
What you don't cover, at all, is an IMO critical new threat that
emerges in the data-plane from the design of the MS protocol.

> If you still think that LISP is using a flow-cache you should have a second
> read to the set of drafts.

This language may appear unclear if you haven't read it in the context
of my other postings.  LISP routing most certainly is a flow-cache,
however, the definition of "flow" is different.  Some platforms and
routing schemes see a flow as a layer-3 destination /32 or similar
(some 90s routers), others more granular (firewalls, where flows are
usually layer-4 and often stateful), and with LISP, the "flow" the
address space routed from your ITR to a remote ETR, which may cover a
large amount of address space and many smaller flows.

The LISP drafts also refer to these flows as "tunnels," but that
language could easily be confused to mean much more permanent, static
tunnels, or MPLS-like tunnels which are signaled throughout the
network of P routers.  So there are clear semantic issues of
importance when talking about LISP, and all these terms must be read
in the correct context.

> For the third time: this is false. We got the problem, we were asking for
> more specific information in order to quantify the risk. We asked you help

You haven't "got it," or you would already understand the risk very
well.  It is not my intention to fault you and your colleagues for
failing to understand this; but to demonstrate clearly that the right
kind of expertise is absolutely not being applied to LISP, and there
is a huge and possibly intractable threat that was completely
overlooked when producing what is meant to be an authoritative
document on currently-known "threats" to LISP.

> to state the problem and explained to you where the solution should be
> addressed. But you seem to be stuck on the operator vs. researcher
> discussion, which IMHO is just pointless.

Substantially all operators are "stuck" there.  They should participate more.

> Let me now ask a simple question: why are you so strongly against LISP?

No new work has been done to address the problem of scaling up the
number of locators or multi-homed end-sites.  However, the *claims*
being made by LISP advocates is that the caching scheme you have,
which is not novel, does solve this problem.  It does not.  It cannot
as there has been no novel work on this.

Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))

[Jeff Wheeler]

On Sun, Jul 17, 2011 at 11:42 AM, William Herrin  wrote:
> My off-the-cuff naive solution to this problem would be to discard the
> oldest incomplete solicitation to fit the new one and, upon receiving
> an apparently unsolicited response to a discarded solicitation,
> restart the process flagging that particular query non-discardable.

Do you mean to write, "flagging that ND entry non-discardable?"  Once
the ND entry is in place, it should not be purged for quite some time
(configurable is a plus), on the order of minutes or hours.  Making
them "permanent" would, however, cause the ND table to eventually
become full when foolish things like frequent source address changes
for "privacy" are in use, many clients are churning in and out of the
LAN, etc.

> Where does this naive approach break down?

It breaks down because the control-plane can't handle the relatively
small number of punts which must be generated in order to send ND
solicits, and without the ability to install "incomplete" entries into
the data-plane, those punts cannot be policed without, by design,
discarding some "good" punts along with the "bad" punts resulting from
DoS traffic.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

_
NANOG mailing list
NANOG@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog

Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Sun, Jul 17, 2011 at 11:07 AM, Eliot Lear  wrote:
> We all make mistakes in not questioning our own positions, from time to
> time.  You, Jeff, seem to be making that very same mistake.

> Rome wasn't built in a day.  The current system didn't come ready-made
> pre-built with all the bells and whistles you are used to.  It grew slowly
> over time, as we learned what works, what doesn't, and what was missing.
> Any system that attempts to deal with locator/id separation will assuredly
> not be built in a day, either.

LISP work has been going on for a long time to still not have any
useful discussion on a designed-in, trivial DoS which will affect any
ITR and make the work being done to allow ETRs to validate source
addresses (or even do loose uRPF) into a DoS vector for ETRs as well.

> While you have stated a problem relating to a security consideration –
> specifically that there is a potential reflection attack that could cause
> cache thrashing, the solution may not be what you expect.

I agree, a solution might be available.  One has not been presented
yet.  In my earliest postings to the IETF LISP list, the ones which
received zero replies, I suggest a way to significantly improve the
cache churn DoS problem.  It is not novel, as Darrel Lewis informed
me, which means that even already-available research has not been
applied to LISP in this area, and the Mapping Service protocol ties
the hands of implementors so they *cannot* apply such techniques while
still conforming to the specifications.

> Yes, you were asked.  Even so... Novelty isn't something worth arguing over,
> except in patent battles.

Really?  Novelty, by definition, advances the state of the art.  You
may not think it's very important to inform people that LISP is based
on essentially the same flow-caching scheme used in the 1990s, but I
do.

> Never is a very long time.  Many uses of "never" have been used relating to
> the Internet.  It is the corollary to "Imminent Death of the 'Net: film @
> 11."  I still have the NANOG tee-shirt with Robert Metcalfe, someone with
> considerably more notoriety, eating his hat.

And yet, I am quite comfortable with the statement that LISP can never
scale up to meet the demands of the Internet.  Perhaps with
fundamental changes to its design, and its advocates giving up some of
their current assumptions, some progress could be made.  In its
current form, though, LISP will never be a useful tool to scale the
Internet, and in fact, it cannot meet the demands of today's Internet.
 Unless, of course, you pretend that the ability to DoS any router
with a trivial amount of traffic is not worthy of concern.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

_
NANOG mailing list
NANOG@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog

Re: NDP DoS attack (was Re: Anybody can participate in the IETF (Was: Why is IPv6 broken?))

[Jeff Wheeler]

On Sun, Jul 17, 2011 at 3:40 PM, Owen DeLong  wrote:
> Basically an ND entry would have the following states and timers:

I've discussed what you have described with some colleagues in the
past.  The idea has merit and I would certainly not complain if
vendors included it (as a knob) on their boxes.  The downfalls of this
approach are that they still don't ensure the discovery of new
neighbors (rather than "ever seen" neighbors) during DoS, and you make
the local DoS a bit more complex by needing to establish more rules
for purging these semi-permanent entries.

> I think most of this punting could be handled at the line card level. Is there
> any reason that the ND process can't be moved into line-card level silicon
> as described above?

You could implement ND solicit in the data-plane (and remove punts
entirely) in even some current chips, to say nothing of future ones.
Whether or not that is a good idea, well, keep in mind that the ND
solicits would then be mcasted to the LAN at a potentially unlimited
rate.

That is not necessarily a problem unless the L2 implementation is not
too good with respect to multicast.  For example, in some "switches"
(mostly those that are routers that can switch) the L2 mcast has
surprising caveats, such as using up a lot of fabric capacity for
whatever replication scheme has been chosen.

Of course, you also hope NDP on all the connected hosts works right.
I believe some Juniper customers noticed a pretty big problem with
JUNOS NDP implementation when deploying boxes using the DE-CIX
addressing scheme, and in a situation like that, the ingress router
for the attack could be crippled by spurious responses from the other
mis-behaving hosts on the LAN, essentially like smurf except without
sending any garbage back out to the Internet.

What you definitely don't want to do is assume this fixes the local
DoS, because it doesn't.  I would like for you to keep in mind that a
host on the LAN, misconfigured to do something like "local proxy-arp,"
or otherwise responding to all ND solicits, would accidentally DoS the
LAN's gateway.  I do not think we should assume that the local DoS
won't happen, or is "fixable" with a whack-a-mole method.

> Sure, that doesn't solve the problem on current hardware, but, it moves it
> from design problem to implementation issue, which IMHO is a step in the
> right direction.

Well, it already is a design problem that implementations can largely
work-around.  Vendors just aren't doing it.  :-/

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

_
NANOG mailing list
NANOG@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog

Re: [lisp] Anybody can participate in the IETF (Was: Why is IPv6 broken?)

[Jeff Wheeler]

On Mon, Jul 18, 2011 at 12:15 PM, Noel Chiappa  wrote:
> Let me make sure I understand your point here. You don't seem to be
> disagreeing with the assertion that for most sites (even things like very
> large universities, etc), their 'working set' (of nodes they communicate)
> with will be much smaller than the network as a whole?

Why would you assume this to be true if LISP also promises to make
multi-homing end-sites cheaper and easier, and independent of the
ISP's willingness to provide BGP without extra cost?  You see, if
every SOHO network and "power user" can suddenly become multi-homed
without spending a great deal of money on a powerful router and ISP
services which support BGP, many of these networks will do so.

The working sets of a scaled-up, LISP future will make the BGP DFZ of
today look small.

> So only the very largest content providers (YouTube, etc) will have
> 'working sets' which include a fairly large share of the entire Internet?

No, any end-site of interest to a DoS attacker must be able to deal
with a working set which includes the entire Internet.  The reason for
this is obvious: it will be the best way to attack a LISP
infrastructure, and it will not be difficult for attackers to send
packets where each packet's source address appears to be from a
different mapping system entry.

Some people have commented that LISP hopes to prevent source address
spoofing through technical means that have not been fully explored.
This is a good goal but it must require the ETR doing address
validation to look-up state from the mapping system.  It will have the
same cache churn problem as an ITR subject to a reflection attack (or
an outbound DoS flow meant to disable that ITR.)

So there is no practical means of doing source address validation on
ETRs (under DoS.)  Even if you did that, the ITR must still be subject
to the occasional large flow of outbound traffic from a compromised
host (dorm machine, open wireless, hacked server, etc.) which is
intended to disable the ITR.

> I have previously commented that such sites have lots of specialized
> infrastructure to handle their traffic loads - do you think it will be
> infeasible for them to have specialized LISP infrastructure too? (Leaving
> aside for a moment what that infrastruture would look like - it's not
> necessarily separate hardware, it might be integrated into existing boxes
> on the periphery of their site.)

Again, every content shop will need to have that specialized
infrastructure.  Every site that someone might have a motive to launch
a DoS attack against must be able to withstand at least trivial DoS.
If you think only the super-huge sites will have a large working set,
you are again ignoring DoS attacks.

The same is true of ISP subscriber access platforms.  If my ISP's BRAS
effectively goes down regularly, I won't keep that ISP service very
long, I'll change to a competitor.  The more subscribers on one BRAS,
the more likely it will receive frequent DoS attacks.

So in reality, the common cache size needed to achieve a high hit rate
really does not matter, unless you wish to ignore DoS (which you seem
to want to do very badly.)

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

_
NANOG mailing list
NANOG@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog

Re: IPv6 end user addressing

[Jeff Wheeler]

On Sat, Aug 6, 2011 at 5:21 AM, Owen DeLong  wrote:
>> At least don't make your life miserable by experimenting with too many 
>> different assignment sizes,
>> or advocate /64s or something, that's considered a design fault which will 
>> come back to you some day.
>> Read the RfCs and RIR policy discussions in the archives some years ago.

Note that in this thread, you advocate three things that are a little
tough to make work together:
* hierarchical addressing plan / routing
* nibble-aligned addressing plan
* minimum /48 per customer

If I were, for example, a hosting company with IPv6 terminated at the
layer-3 ToR switch, I would then use a /40 per rack of typical
"dedicated servers."  If you then want some bits to be a POP-locator
field for your hierarchical routing scheme, you are already forced to
request more than a /32.  The number of customers per layer-3 device
for typical end-user access networks was around the same into the
late-1990s/early-2000s, as ISPs had racks of Portmasters or whatever
box of choice for terminating dial-up.

Densities have changed, but this doesn't necessarily win you an
advantage when combining those three properties.  This is especially
true if you consider that density may change in a difficult-to-predict
manner in the future -- a BRAS box with a couple thousand customers
today might have three times as many in a couple of years (IPv6 is
supposed to help us avoid renumbering or injecting additional routes
into our network, right?)  As an access provider, if I shared your
view, I would be reserving a /36 or /32 per BRAS box.  If I then want
some additional bits for hierarchical routing ... I'm going to need a
pretty large address block for perhaps a pretty small number of
customers.  After all, my scheme, applying your logic, dictates that I
should use a /32 or perhaps a /28 per each POP or city (I need to plan
for several BRAS each), even if I don't have a lot of customers today!

I think /56 is more sensible than /48, given the above, for most
end-users.  Either way, the users will be gaining a lot more
flexibility than they have with IPv4 today, where they probably get
just one IP address and have to pay a fee for any extras.  Giving the
typical end-user 8 fewer bits worth of address space allows the ISP
network more flexibility for hierarchical routing before they have to
go to their RIR and figure out how to justify an out-sized allocation.

Also, if folks would stop thinking that every subnet should be a /64,
they will see that end-users, makers of set-top-gateways, or whatever,
can certainly address a whole lot of devices in a whole lot of subnets
even if the user is only given a /64.  Do we think DHCPv6 won't be the
most common way of assigning addresses on SOHO LANs, and that SLAAC
will be essential?  I, for one, think that some ISPs will be sick and
twisted enough to hand out /128s so they can continue charging for
more IP addresses; but certainly the makers of IPv6-enabled devices
will foresee that end-user LANs might not be /64 and include the
necessary functionality to work correctly with smaller subnets.

Before you beat me to it, yes, we seem to have completely opposing
views on this subject.  I will change my mind when I can go to the RIR
and get a IPv6 /24 for a small ISP with a few POPs and a few tens of
thousands of customers.  Should RIR policy permit that sort of thing?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Sat, Aug 6, 2011 at 12:36 PM, Owen DeLong  wrote:
> On Aug 6, 2011, at 3:15 AM, Jeff Wheeler wrote:
>> Note that in this thread, you advocate three things that are a little
>> tough to make work together:
>> * hierarchical addressing plan / routing
>> * nibble-aligned addressing plan
>> * minimum /48 per customer
>>
> Hasn't been hard so far.

Well, you aren't actually doing this on your network today.  If you
practiced what you are preaching, you would not be carrying aggregate
routes to your tunnel broker gateways across your whole backbone.
Perhaps you also wouldn't use one allocation on the tunnel broker
gateway for /64s and another, a /37 in the case of Ashburn for
example, for users who self-provision a /48 from it.  Also, of course,
your default assignment plan is /64, not /48, even though there are
typically around, what, 10k tunnels active network-wide?

To be clear, you don't do any of the three things you advocate above
where Hurricane Electric's tunnel broker service is concerned.

> I think we were talking about access customers. I don't see giving /48s
> to individual dedicated servers as I don't see them having hierarchy
> behind them. I would think that a /48 per TOR switch would be
> reasonable in that case.

I wish there was more discussion about IPv6 addressing plans in
hosting environments on this list.  I think that, rarely, customers
will decide to tunnel from their home or office to their "dedicated
server," co-lo rack, etc.  My addressing policies provide for this
type of customer to receive a /56 upon request without breaking my
hierarchical addressing scheme.  If they need more than that, the
layer-3 aggregator has to inject an additional route into the POP
area.  If a whole bunch of customers on one aggregator ask for /56,
then the aggregator needs an extra /48 (which might really mean
growing its existing /48 to a shorter route.)

> However, requesting more than a /32 is perfectly reasonable. In
> the ARIN region, policy 2011-3.

My read of that policy, and please correct me if I misunderstand, is
that it recognizes only a two-level hierarchy.  This would mean that
an ISP could use some bits to represent a geographic region, a POP, or
an aggregation router / address pool, but it does not grant them
justification to reserve bits for all these purposes.

> density, even in 20 years. I realize that customer density in
> urban areas does tend to increase, but, assuming a maximum
> 50% market penetration, serving a city the size of Philadelphia
> out of a single POP still seems unlikely to me.

AT&T serves some entire states out of a single POP, as far as layer-3
termination is concerned.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Sat, Aug 6, 2011 at 7:26 PM, Owen DeLong  wrote:
>> Well, you aren't actually doing this on your network today.  If you
>> practiced what you are preaching, you would not be carrying aggregate
>> routes to your tunnel broker gateways across your whole backbone.
>
> Yes we would.

No, if you actually had a hierarchical addressing scheme, you would
issue tunnel broker customer /64s from the same aggregate prefix that
is used for their /48s.  You'd then summarize at your ABRs so the
entire POP need only inject one route for customer addressing into the
backbone.  Of course, this is not what you do today, and not because
of limited RIR allocation policies -- but because you simply did not
design your network with such route aggregation in mind.

> Those are artifacts of a small allocation (/32) from a prior RIR policy.
> The fact that those things haven't worked out so well for us was one of
> the motivations behind developing policy 2011-3.

There was nothing stopping you from using one /48 out of the /37s you
use to issue customer /48 networks for issuing the default /64 blocks
your tunnel broker hands out.

> We give a minimum /48 per customer with the small exception that
> customers who only want one subnet get a /64.

You assign a /64 by default.  Yes, customers can click a button and
get themselves a /48 instantly, but let's tell the truth when talking
about your current defaults -- customers are assigned a /64, not a
/48.

> We do have a hierarchical addressing plan. I said nothing about routing,
> but, we certainly could implement hierarchical routing if we arrived at a
> point where it was advantageous because we have designed for it.

How have you designed for it?  You already missed easy opportunities
to inject fewer routes into your backbone, simply by using different
aggregate prefixes for customer /64s vs /48s.

>>> However, requesting more than a /32 is perfectly reasonable. In
>>> the ARIN region, policy 2011-3.
>>
>> My read of that policy, and please correct me if I misunderstand, is
>> that it recognizes only a two-level hierarchy.  This would mean that
>> an ISP could use some bits to represent a geographic region, a POP, or
>> an aggregation router / address pool, but it does not grant them
>> justification to reserve bits for all these purposes.
>>
>
> While that's theoretically true, the combination of 25% minfree ,
> nibble boundaries, and equal sized allocations for all POPs based
> on your largest one allows for that in practical terms in most
> circumstances.

I don't think it does allow for that.  I think it requires you to have
at least one "POP prefix" 75% full before you can get any additional
space from the RIR, where 75% full means routed to customers, not
reserved for aggregation router pools.  This is not a hierarchy, it is
simply a scheme to permit ISPs to bank on having at least one level of
summarization in their addressing and routing scheme.

2011-3 does not provide for an additional level to summarize on the
aggregation routers themselves.  It should, but my read is that the
authors have a very different opinion about what "hierarchical"
addressing means than I do.  It should provide for route aggregation
on both the ABR and the aggregation router itself.

>> AT&T serves some entire states out of a single POP, as far as layer-3
>> termination is concerned.
>>
>
> Are any of the states with populations larger than Philadelphia among
> them?

Yes, for example, Indiana.  Pretty much every state in the former
Ameritech service territory.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Sun, Aug 7, 2011 at 6:58 PM, Mark Andrews  wrote:
> So you want HE to force all their clients to renumber.

No.  I am simply pointing out that Owen exaggerated when he stated
that he implements the following three practices together on his own
networks:
* hierarchical addressing
* nibble-aligned addressing
* /48 per access customer

You can simply read the last few messages in this thread to learn that
his recommendations on this list are not even practical for his
network today, because as Owen himself says, they are not yet able to
obtain additional RIR allocations.  HE certainly operates a useful,
high-profile tunnel-broker service which is IMO a very great asset to
the Internet at-large; but if you spend a few minutes looking at the
publicly available statistics on this service, they average only
around 10,000 active tunnels across all their tunnel termination boxes
combined.  They have not implemented the policies recommended by Owen
because, as he states, a /32 is not enough.

Do I think the position he advocates will cause the eventual
exhaustion of IPv6?  Well, let's do an exercise:

There has been some rather simplistic arithmetic posted today, 300m
new subnets per year, etc. with zero consideration of address/subnet
utilization efficiency within ISP networks and individual aggregation
router pools.  That is foolish.  We can all pull out a calculator and
figure that 2000::/3 has space for 35 trillion /48 networks.  That
isn't how they will be assigned or routed.

The effect of 2011-3 is that an out-sized ISP like AT&T has every
justification for deciding to allocate 24 bits worth of subnet ID for
their "largest POP," say, one that happens to terminate layer-3
services for all customers in an entire state.  They then have policy
support for allocating the same sized subnet for every other POP, no
matter how small.  After all, the RIR policy permits them to obtain
additional allocations as soon as one POP subnet has become full.

So now you have a huge ISP with a few huge POPs, and a lot of small
ones, justified in assigning the same size aggregate prefix, suitable
for 2^24 subnets, to all those small POPs as well.  How many layer-3
POPs might this huge ISP have?  Any number.  It could be every central
office with some kind of layer-3 customer aggregation router.  It
could even be every road-side hut for FTTH services.  Perhaps they
will decide to address ten thousand POPs this way.

Now the nibble-aligned language in the policy permits them to round up
from 10,000 POPs to 16 bits worth of address space for "POP ID."  So
AT&T is quite justified in requesting:
48 (customer subnet length) - 24 (largest POP subnet ID size) - 16
(POP ID) == a /8 subnet for themselves.

Now you can see how this policy, and addressing scheme, is utterly
brain-dead.  It really does put you (and me, and everyone else) in
real danger of exhausting the IPv6 address space.  All it takes is a
few out-sized ISPs, with one large POP each and a bunch of smaller
ones, applying for the maximum amount of address space permitted them
under 2011-3.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Wed, Aug 10, 2011 at 6:55 AM, Alexander Harrowell
 wrote:
> Thinking about the CPE thread, isn't this a case for bridging as a
> feature in end-user devices? If Joe's media-centre box etc would bridge
> its downstream ports to the upstream port, the devices on them could
> just get an address, whether by DHCPv6 from the CPE router's delegation
> or by SLAAC, and then register in local DNS or more likely do multicast-
> DNS so they could find each other.

This would require the ISP gateway to have IPv6 ND entries for all of
the end-user's devices.  If that is only a few devices, like the
typical SOHO LAN today, that's probably fine.  It is not fine if I
purchase some IPv6-connected nanobots.  Given today's routers, it is
probably not even fine if the average SOHO goes from 1 state entry to
just 20 or 30.  I have about 20 devices in my home that use the
Internet -- TVs, DVRs, VoIP telephones, printer, mobile phones with
Wi-Fi, a couple of video game consoles, etc.  I imagine that is not
atypical these days.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Wed, Aug 10, 2011 at 2:03 PM, Owen DeLong  wrote:
> That said, /48 to the home should be what is happening, and /56 is
> a better compromise than anything smaller.

Is hierarchical routing within the SOHO network the reason you believe
/48 is useful?  You don't really imagine that end-users will require
more than 2^8 subnets, but that they will want several levels of very
simple, nibble-aligned routers within their network?

This is perhaps a good discussion to have.  I, for one, see CPE
vendors still shipping products without IPv6 support at all, let alone
any mechanism for creating an address or routing hierarchy within the
home without the end-user configuring it himself.  I am not aware of
any automatic means to do this, or even any working group trying to
produce that feature.

Is it true that there is no existing work on this?  If that is the
case, why would we not try to steer any such future work in such a way
that it can manage to do what the end-user wants without requiring a
/48 in their home?

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Wed, Aug 10, 2011 at 7:12 PM, Owen DeLong  wrote:
>> Is it true that there is no existing work on this?  If that is the
>> case, why would we not try to steer any such future work in such a way
>> that it can manage to do what the end-user wants without requiring a
>> /48 in their home?
>
> No, it is not true.

Can you give any example of a product, or on-going work?  I have read
two posts from you today saying that something either exists already,
or is being worked on.  I haven't read this anywhere else.

> I suppose that limiting enough households to too small an allocation
> will have that effect. I would rather we steer the internet deployment
> towards liberal enough allocations to avoid such disability for the
> future.
>
> Have we learned nothing from the way NAT shaped the (lack of)
> innovation in the home?

I am afraid we may not have learned from exhausting IPv4.  If I may
use the Hurricane Electric tunnel broker as an example again,
supposing that is an independent service with no relation to your
hosting, transit, etc. operations, it can justify a /24 allocation
immediately under 2011-3, without even relying on growth projections.
That's a middle ground figure that we can all live with, but it is
based on you serving (at this moment) only 8000 tunnels at your
busiest tunnel gateway.  If your tunnel gateways could serve 12,288 +
1 users each, then your /24 justification grows to a /20.  So you
would have a pretty significant chunk of the available IPv6 address
space for a fairly small number of end-users -- about 72,543 at
present.

It isn't hard to do some arithmetic and guess that if every household
in the world had IPv6 connectivity from a relatively low-density
service like the above example, we would still only burn through about
3% of the IPv6 address space on end-users (nothing said about server
farms, etc. here) but what does bother me is that the typical end-user
today has one, single IP address; and now we will be issuing them 2^16
subnets; yet it is not too hard to imagine a future where the global
IPv6 address pool becomes constrained due to service-provider
inefficiency.

I would like to have innovations in SOHO devices, too; who knows what
these may be.  But I fear we may repeat the mistake that caused NAT to
be a necessity in IPv4 -- exhausting address space -- by foolishly
assuming that every household is going to need twenty-four orders of
magnitude more public addresses than it has today.

That is what these practices do -- they literally give end-users
twenty-four orders of magnitude more addresses, while it is easy to
imagine that we will come within one order of magnitude of running
completely out of IPv6 addresses for issuing to service providers.

I didn't know what the digit "1" followed by twenty-four zeroes was
called.  I had to look it up.  So our end-users will be receiving
about one-Septillion addresses to use in their home, but no one seems
to be asking what future technology we may be harming by possibly
constraining the global address pool.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 end user addressing

[Jeff Wheeler]

On Wed, Aug 10, 2011 at 8:40 PM, Mark Andrews  wrote:
> No.  A typical user has 10 to 20 addresses NAT'd to one public address.

I'd say this is fair.  Amazingly enough, it all basically works right
with one IP address today.  It will certainly be nice to have the
option to give all these devices public IP addresses, or even have a
few public subnets; but it does require more imagination than any of
us have demonstrated to figure out how any end-user will need more
than 2^8 subnets.  That's still assuming that device-makers won't
decide they need to be able to operate with subnets of arbitrary size,
rather than fixed-size /64 subnets.

> There was a concious decision made a decade and a half ago to got to
> 128 bits instead of 64 bits and give each subnet 64 bits so we would
> never have to worry about the size of a subnet again.  IPv6 is about
> managing networks not managing addresses.

Thanks for the explanation of how to subnet IPv4 networks and use
RFC1918.  I hope most readers are already familiar with these
concepts.  You should note that IPv6 was not, in fact, originally
envisioned with /64 subnets; that figure was to be /80 or /96.  In the
mid-1990s, it was believed that dramatically increasing the number of
bits available for ISP routing flexibility was very beneficial, as
well as making access subnets so big that they should never need to
grow.  Then SLAAC came along.  Except SLAAC doesn't do necessary
things that DHCPv6 does, and the cost of implementing things like
DHCPv6 in very small, inexpensive devices has gone down dramatically.

I am amazed that so few imagine we might, in within the lifetime of
IPv6, like to have more bits of address space for routing structure
within ISP networks; but these people do think that end-users need
1.2e+24 addresses for the devices they'll have in their home.

I don't have to use my imagination to think of ways that additional
bits on the network address side would have been advantageous -- all I
need is my memory.  In the 90s, it was suggested that a growing number
of dual-homed networks cluttering the DFZ could be handled more
efficiently by setting aside certain address space for customers who
dual-homed to pairs of the largest ISPs.  The customer routes would
then not need to be carried by anyone except those two ISPs, who are
earning money from the customer.  This never happened for a variety of
good reasons, but most of the technical reasons would have gone away
with the adoption of IPv6, as it was envisioned in the mid-90s.

There seems to be a lot of imagination being used for SOHO networks,
and none on the ISP side.  What a shame that is.

Owen, I do agree with the point you made off-list, that if huge
mistakes are made now and the IPv6 address space is consumed more
rapidly than the community is comfortable with, there should be plenty
of opportunity to fix that down the road.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: OSPF vs IS-IS

[Jeff Wheeler]

I thought I'd chime in from my perspective, being the head router
jockey for a bunch of relatively small networks.  I still find that
many routers have support for OSPF but not IS-IS.  That, plus the fact
that most of these networks were based on OSPF before I took charge of
them, in the absence of a compelling reason to change to another IGP,
keeps me from taking advantage of IS-IS.  I'd like to, but not so
badly that I am willing to work around those routers without IS-IS, or
weight that feature more heavily when purchasing new equipment.

There are many routers with OSPF but no IS-IS.  I haven't seen any
with IS-IS but no OSPF.  I don't think such router would be very
marketable to most non-SP networks.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Deploying IPv6 Responsibly

[Jeff Wheeler]

On Fri, Aug 19, 2011 at 12:59 PM, Frank Bulk  wrote:
> I just noticed that the quad-A records for both those two hosts are now
> gone.  DNS being what it is, I'm not sure when that happened, but our
> monitoring system couldn't get the  for www.qwest.com about half an hour
> ago.
>
> Hopefully CenturyLink is actively working towards IPv6-enabling their sites
> again.

I hope that they aren't.  It doesn't help anyone for Qwest/CenturyLink
to publish  records or otherwise activate IPv6 services if they
have no system for monitoring their single most publicly-visible
service, no mechanism for alerting engineers or system administrators
of trouble, no way to act on problem reports generated by users after
*ten days*, and apparently no ability to actually fix the problem in a
timely manner when someone with a clue finally realized what was going
on.

Let's not encourage Qwest, or anyone else, to deploy any more IPv6
services until they get a few things in order first.  Simply turning
the  record back on before major, systemic oversights within the
organization are fixed would be irresponsible.  It will not help IPv6
progress, it will hurt it.

Every other network should keep this in mind as well.  If you can't
support your IPv6 services, don't deploy them for public use yet!
This doesn't mean don't work on it, but if your tech support staff
don't know how to handle calls, if the workstations in your
call-center don't have IPv6, if you haven't trained every person on
the escalation tree -- publishing an  record for www.foo.com is a
pretty stupid thing to do.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: iCloud - Is it going to hurt access providers?

[Jeff Wheeler]

On Sun, Sep 4, 2011 at 4:45 PM, Wayne E Bouchard  wrote:
> Okay, so to state the obvious for those who missed the point...
>
> The congestion will either be directly in front of user because
> they're flooding their uplink or towards the destination (beit a
> single central network or a set of storage clusters housed at, say, 6
> different locations off 3 different providers.) It is very hard, in my

If scaling up Internet bandwidth were the hardest thing about
deploying SaaS / "cloud" services, don't you think transit vendors
would suddenly be more profitable than EMC and friends?  It should be
obvious to you, and everyone else, that datacenter Internet
connectivity is a trivial concern compared to everything else that
goes into these platforms.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: BGP conf

[Jeff Wheeler]

On Tue, Nov 1, 2011 at 9:01 PM, Edward avanti  wrote:
> many example seem
> insecure no prefix list so on.
...
> I am not ignorant with cisco 7201, but am total newby to BGP.

Your concern about a lack of any prefix-lists in the documentation /
examples you have read is justified.  If you are connecting to an IX
it may offer route-servers which have prefix-lists maintained by the
IX staff and tools.  However, as you may already know, you will only
receive the "best path" to each prefix from an IX route-server.  This
is often a motive (among others) to establish direct eBGP sessions
with other IX members.  Once you start doing that, you had better
filter routes from those neighbors, or you will subject your network
to your peers' mistakes and glitches.

If you imagine that the IX has other members like yourself, who also
do not know much about BGP, then you can understand why you do not
want your peers' mistakes to cause outages on your network.

Doing a "cut, replace, and paste" from online examples is obviously a
bad idea.  If I were you, I would find a local consultant (perhaps
someone on the staff of the IX or another member) who can assist you
with your initial configuration, and help you in the event of a severe
emergency.  Otherwise, frankly, you are going to be better off by just
buying transit from Verizon and being single-homed.  The added
complexity of BGP is not an asset to an organization that doesn't have
adequate expertise.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: BGP conf

[Jeff Wheeler]

On Wed, Nov 2, 2011 at 7:50 PM, Edward avanti  wrote:
> sorry, my english not so perfect, at no time I mean send to IX what Verizon
> send me, I'm not THAT stupid hehe
> I mean if destination/origin is via IX, then send THAT traffic only by IX
> and not Verizon.

I understood what you mean.  The recommendations in my earlier reply
are still the best ones you've received:
1) hire a consultant to assist you both now and with any future problems
or 2) do not worry about being multi-homed, because the extra
complexity will do you more harm than good

Imagine if you took your car to a shop and asked for new tires, and
the mechanic said, "well, I have never changed tires before and I'm
not sure I have the right tools, but if you give me a couple of days I
think I can read about it on the Internet and figure it out."  Of
course you would not buy tires from him, you would go to another shop.
 That mechanic would quickly find that, if he wants to sell tires, he
needs to learn how to install them or hire someone to do it for him.

What you are asking your boss/company to do is trust you to put tires
on their car without the right tools or knowledge.  The result of that
is probably how your network will end up: "a wreck."

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: BGP conf

[Jeff Wheeler]

On Wed, Nov 2, 2011 at 8:44 PM, Jack Bates  wrote:
> Now I have the mile long monstrosity that uses BGP communities for
> everything, and of route-maps/policies with prefix-lists for downstream
> customers. You have to start somewhere.
>
> cymru secure bgp templates is probably a good beginning.

I guess ten years of watching RIRs and users de-bogon new /8s didn't
teach you why those Cymru examples are more dangerous than they are
good.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: BGP conf

[Jeff Wheeler]

On Wed, Nov 2, 2011 at 10:04 PM, Jack Bates  wrote:
> Have to read the current cymru bgp templates?
>
> ! manner. Why not consider peering with our globally distributed bogon
> ! route-server project? Alternately you can obtain a current and well

I'm not telling you something you don't already know, but for the
novices who regard this list as a source of expertise, I will explain
in greater detail why this is a really dumb idea.

If you took a list of bogons over eBGP from Cymru, you would get
unused /8s and similar.  What you don't get is a route that matches
whatever silly thing someone on the DFZ accidentally leaked: a
more-specific that will still cause you to route traffic to their
leaked prefix out to the Internet (and presumably, to their network.)

There is nothing good about this.  It's just adding unnecessary
complexity for no operational benefit.  There is bad about it.  It
adds complexity and risk.  What is that risk?  If you decide that the
Cymru "distributed bogon route-server" is for you, and simply rewrite
next-hops received on that session to Null0, it is possible that Cymru
could make an error, or otherwise introduce non-bogon routes into your
network as if they were bogons, causing black-holes.  This is
obviously too much to risk for something that has no operational
benefit.

The Cymru guys do many positive things.  One of the more questionable
things they do, though, is operate a route-server with the intention
of black-holing botnet C&C IPs on a very wide scale.  This is
certainly a positive thing to do, but it was not done in a transparent
manner; and in fact didn't even have management approval at Cogent
when they configured it on their network.  There was no established
channel to find out why your IP address appeared on this list or to
get it removed.  All it took for me to get the whole idea canned at
Cogent was one inquiry to management, asking why engineers had quietly
started using a clandestine blackhole list operated by a third-party
and would not give any answers to a customer if one of their IPs
appeared on that list.  The IP address I inquired about was certainly
not a botnet C&C node, and how it ended up on that list is a mystery.
I'm not saying there was any malicious intent, but it was a mistake at
least.

Trusting that "bogon" black-hole list to do something you don't even
need to do anyway is not smart.  It's *especially* not smart for some
novice who doesn't understand the implications of his decision.  This
is the danger of "cut & paste engineering."

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Anyone seen this kind of problem? SIP traffic not getting to destination but traceroute does

[Jeff Wheeler]

On Wed, Nov 9, 2011 at 1:47 PM, Jay Nakamura  wrote:
> So my questions is, is it possible there is some kind of filter at
> Qwest or Level 3 that is dropping traffic only for udp 5060 for select
> few IPs?  That's the only explanation I can come up with other than

I ran into exactly this problem last week with Rogers.  All traffic
from the client except udp/5060 could be received by us, and udp/5060
was blocked.  We tested other IP addresses on our (provider) side and
did not find any blocking there, so we assigned a new IP to the SIP
gateway.  I hardly think this can be an ordinary malfunction, but good
luck getting a phone company to troubleshoot a problem with their
subscribers using mobile data to connect to a third-party voice
gateway...

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Mon, Nov 28, 2011 at 4:51 PM, Owen DeLong  wrote:
> Technically, absent buggy {firm,soft}ware, you can use a /127. There's no
> actual benefit to doing anything longer than a /64 unless you have
> buggy *ware (ping pong attacks only work against buggy *ware),
> and there can be some advantages to choosing addresses other than
> ::1 and ::2 in some cases. If you're letting outside packets target your
> point-to-point links, you have bigger problems than neighbor table
> attacks. If not, then the neighbor table attack is a bit of a red-herring.

Owen and I have discussed this in great detail off-list.  Nearly every
time this topic comes up, he posts in public that neighbor table
exhaustion is a non-issue.  I thought I'd mention that his plan for
handling neighbor table attacks against his networks is whack-a-mole.
That's right, wait for customer services to break, then have NOC guys
attempt to clear tables, filter traffic, or disable services; and
repeat that if the attacker is determined or going after his network
rather than one of his downstream customers.

I hate to drag a frank, private discussion like that into the public
list; but every time Owen says this is a non-issue, you should keep in
mind that his own plan is totally unacceptable for any production
service.  Only one of the following things can be true: either 1) Owen
thinks it is okay for services to break repeatedly and require
operator intervention to fix them if subjected to a trivial attack; or
2) he is lieing.  Take that as you will.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Tue, Nov 29, 2011 at 1:43 AM,   wrote:
> It's worked for us since 1997.  We've had bigger problems with IPv4 worms

That's not a reason to deny that the problem exists.  It's even
fixable.  I'd prefer that vendors fixed it *before* there were massive
botnet armies with IPv6 connectivity, but in case they don't, I do not
deploy /64.

On Tue, Nov 29, 2011 at 2:20 AM, Jonathan Lassoff  wrote:
> Agreed. While I don't have any good numbers that I can publicly offer up, it
> also intuitively makes sense that there's a greater proportion of IPv4 DDOS
> and resource exhaustion attacks vs IPv6 ones.

Of course.  There are comparably few hosts with IPv6 connectivity.
Bad guys aren't that familiar with IPv6 yet.  Even if they are, their
armies of compromised desktops probably can't launch an effective IPv6
attack yet.  Lack of sources, no way to get nasty IPv6 packets to the
target, or the target has different infrastructure for IPv4 and IPv6
anyway, and taking out the IPv6 one only isn't that beneficial (Happy
Eyeballs features and such.)

Further, the victim can just turn off IPv6 when they start getting
attacked in this way.  And that is exactly what sites will end up
doing, turning off IPv6 because vendors aren't addressing issues like
these.  That doesn't help anyone.

> I imagine the mitigation strategies are similar for both cases though: just
> rate-limit how often your router will attempt neighbor discovery. Are there
> other methods?

Simply rate-limiting the data-plane events that trigger ND resolution
is not good enough.  One very popular platform that is offered with
cards in horizontal or vertical orientation uses the same policer for
ARP and NDP.  That means when you do eventually start getting ND
attacks, it will break your IPv4 services also.

If you want to learn more about this, I have some slides:
http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Tue, Nov 29, 2011 at 12:42 AM, Owen DeLong  wrote:
> That's _NOT_ a fair characterization of what I said above, nor is it
> a fair characterization of my approach to dealing with neighbor table
> attacks.

Here are some direct quotes from our discussion:
> Since we have relatively few customers per aggregation router, I don't
> think you'd be nearly as successful as you think you would.

> On the platforms we use, it won't spill over into IPv4 breakage. Additionally,
> it will break fewer than 50 other dedicated servers and no other customers.
> This will be tolerated for about 5 minutes while our support department
> receives the alarm and looks into the cause, then, your dedicated server
> won't have power any more and the problem will go away along with your
> account.

At no time did you ever suggest you had any idea how to systemically
solve this problem.  Now you are claiming that you have a "more
global" approach, but this is another of your lies.

> All of our support guys have enough clue to get to this quickly enough
> and our monitoring systems would detect the abnormally large
> neighbor table fairly early in the process.

Your monitoring systems keep an eye on the size of your ND tables?
How can this be true if you believe that ND attacks are not an issue?
Did you just throw resources at monitoring this for no reason?  Do you
really even poll or alert on this data, or were you simply telling
another lie?

> Additionally, we have a network engineer on duty 24x7, so, even
> if the support guys don't figure it out correctly, there's backup with
> clue right behind them in the same room.

That is exactly "NOC whack-a-mole."

> What I said above is that if you allow random traffic aimed at your
> point to point links, you're doing something dumb.

If your network has nothing but point-to-point links, it is easy to
defend.  Sadly that is not how you or I interface with many of our
customers.

In addition, you don't actually practice what you preach.  Hurricane
Electric uses /126 networks in its backbone.  Why are you not rushing
to change these to /64?  After all, you regularly tell us about the
supposed disadvantages of /126 on point-to-point links.  What are
these disadvantages?

> As to my actual plan for dealing with it, what I said was that if we
> ever see a neighbor table attack start causing problems with services,
> we'd address it at that time. Likely we would address it more globally
> and not through a whack-a-mole process.

No, this is not what you said.  Again, you are simply telling lies.

> I did not give details of all of our mitigation strategies, nor can I.

Yes you did.  Your "strategy" is whack-a-mole.

> What I can say is that we do have several /64s that could be attacked
> such that we'd notice the attack. Most likely the attack wouldn't break
> anything that is a production service before we could mitigate it.

Breaking about 50 customers for as long as it takes your support staff
or NOC to troubleshoot, in your mind, muts not be "breaking anything
that is a production service," or else "before we could mitigate it"
means you have figured out how to travel through time.

> In more than a decade of running production IPv6 networks, we have
> yet to see a neighbor table attack. Further, when you consider that
> most attacks have a purpose, neighbor table attacks are both more
> difficult and less effective than other attack vectors that are readily
> available. As such, I think they are less attractive to would-be attackers.

Again, the "bad guys" don't have much motive (yet) since few services
of interest share common IPv4 and IPv6 infrastructure today.  That
will change.

> No, there is a third possibility.
>
> I don't mind you taking a frank private discussion public (though
> it's not very courteous), but, I do object to you misquoting me
> and misconstruing the nature and substance of what I said.

It's disingenuous of you to continue to lie every time this topic
comes up on the mailing list.

> Yes, ND attacks are possible if you leave your /64 wide open to
> external traffic. However, if you're using your /64 to provide services,
> chances are it's pretty easy to cluster your server in a much smaller
> range of addresses. A simple ACL that only permits packets to
> that range (or even twice or 4 times that range) will effectively
> block any meaningful ND attack.
>
> For example, let's say you use 2001:db8:fe37:57::1000:0
> to 2001:db8:fe37:57:1000:01ff as the IPv6 range for a
> set of servers.
>
> Let's say there are 200 servers in that range.
>
> That's 200/512 good ND records for servers and 312 slots
> where you can put additional servers. That gives you a total
> attack surface of 312 incomplete ND records.
>
> This was part of my frank private discussion with Jeff, but,
> he seems to have forgotten it.

Since I've re-read our earlier discussion (unlike you) I can state
with certainty that it was not part of our earlier discussion.  If it
was, I would be happy t

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Wed, Nov 30, 2011 at 9:48 AM, Ray Soucy  wrote:
> 1. Using a stateful firewall (not an ACL) outside the router
> responsible for the 64-bit prefix.  This doesn't scale, and is not a
> design many would find acceptable (it has almost all the problems of
> an ISP running NAT)

Owen has suggested "stateful firewall" as a solution to me in the
past.  There is not currently any firewall with the necessary features
to do this.  We sometimes knee-jerk and think "stateful firewall has
gobs of memory and can spend more CPU time on each packet, so it is a
more likely solution."  In this case that does not matter.  You can't
have 2^64 bits of memory.

You could make a firewall with the needed features (or a layer-3
switch), but it would have to be the layer-3 gateway of the subnets
you are protecting (not an upstream device) and it would need
knowledge of all addresses in use on the subnet, which must fit within
its ND table limits.  Only DHCP snooping can do this and customers are
not exactly keen on receiving DHCP-assigned addresses in mixed
datacenter environments, even if the addresses are static ones.  Once
you do that, you need to limit the number of addresses that can be
leased to each customer to far less than a /64 anyway.  All you gain
by having all that complexity is the appearance of bigger subnets,
when in reality, they are non-functional except for the limited number
of addresses which are actively leased out.

Again the arguments for /64 are not promising.  It is much less
complicated to simply deploy a longer subnet.

On Wed, Nov 30, 2011 at 11:13 AM, Jimmy Hess  wrote:
> On Wed, Nov 30, 2011 at 8:48 AM, Ray Soucy  wrote:
>> Saying you can mitigate neighbor table exhaustion with a "simple ACL"
>> is misleading (and you're not the only one who has tried to make that
>> claim).
>
> It's true, though, you can.
>
> From a network design POV, there may still be reasons to prefer the ACL 
> method.
> They better be good reasons, such as a requirement for SLAAC on a large LAN.

No, Jimmy, you can't do that with SLAAC.  I do not think you
understand the problem.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Wed, Nov 30, 2011 at 3:13 PM, Owen DeLong  wrote:
> As such, I prefer to deploy IPv6 as it is today and resolve the bugs
> and the security issues along the way (much like we did with IPv4).

Why is the Hurricane Electric backbone using /126 link-nets, not /64?
You used to regularly claim there are significant disadvantages to
longer subnets.  At best, you are still claiming there are no
advantages.  These are lies.  Please, Owen, tell us why you aren't
practicing what you preach.

> I haven't said that security issues should be ignored, either. Just that
> they should be viewed in a proper context and assessed with a realistic
> evaluation of the magnitude of the risk and the difficulty of mitigation.

You repeatedly claim that ND exhaustion is a non-issue.  You also
claim you have secret sauce to mitigate attacks.  This, after you
previously claimed that you were using common ACLs to mitigate
attacks, and I showed you how that cannot be true.  Your understanding
of this problem has rocketed from totally clueless to having secrets
you can't discuss.  Except it isn't, because you are also advocating
... denying all traffic to all subnets except the first few hundred
addresses.  What a stellar plan!

Just stop telling lies about this, Owen.  That's all I'm asking.

You, personally, are part of the problem.  If the guy who is supposed
to be the public-facing technical outreach guy for the self-described
leader in IPv6 transit/hosting/etc services continues to go around
claiming this is a non-issue, when it very clearly is, that is
destructive, not helpful.

> What has also been lost here is that my description of the various
> mitigation tactics for ND exhaustion attacks depends on the type
> of network being protected. Strategies that work for point-to-point
> links (simple ACLs at the borders in most environments, for
> example) are not the same as strategies that work to protect
> client LANs (stateful firewalls with default deny inbound) or
> strategies necessary to protect server LANs (slightly more complex
> ACLs and other tactics).

You have no such "simple ACLs at the borders" on the Hurricane
Electric network.  In fact, your mitigation mechanism for the backbone
is exactly what I recommend: deploy longer subnets.  You don't have
any mitigation mechanism for your hosting services, other than
whack-a-mole.

If anyone has trouble believing me, you can do what I did, and email
Owen off-list.  You can say, Owen, I'd like to subscribe to a
Hurricane Electric dedicated server, get myself a /64, and DoS my own
subnet, to see if that affects my box or any other nearby customers.
The reply you'll get will be that your box will be powered off,
because they have no mitigation strategy.

Arguing in the abstract is all fun and games, but when you ask Owen to
show you something that works in a real-world, production environment,
he can't.  That's because Owen's network design is not suitable for
production use in his own environment with routers he claims to have
selected in part based on their performance under ND attacks (another
lie.)

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Link local for P-t-P links? (Was: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?)

[Jeff Wheeler]

On Wed, Nov 30, 2011 at 9:15 PM, Mike Jones  wrote:
> Link-Local?
>
> For "true" P-t-P links I guess you don't need any addresses on the

Point-to-point links in your backbone are by far the easiest thing to
defend against this attack.  I wish we would steer the discussion away
from point-to-point links that are entirely within the control of the
operator, as this is really quite well understood.  Major ISPs
including Level3 are already doing /126 to their customers today as
well.  In fact, Level3 does not even reserve a /64, they will hand out
::0/126 to one customer on a given access router, ::4/126 to the next.
 It clearly works.

The access layer for non point-to-point customers, on the other hand,
is less well-understood.  That's why we keep having these discussions.
 Getting customers (and their device/software) to work correctly with
link-local addressing and DHCP-PD or similar is going to be an uphill
battle in a hosting environment.  It also breaks down immediately if
the hosting customer, for example, wishes to use ND to be able to
provision addresses on two or more servers from a common subnet.  So
there are both perception and practical problems / limitations with
this approach.  I'm not saying it's a bad idea, but it won't work in
some instances.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 prefixes longer then /64: are they possible in DOCSIS networks?

[Jeff Wheeler]

On Thu, Dec 1, 2011 at 9:42 AM, Chuck Anderson  wrote:
> Jumping in here, how about static ND entries?  Then you can use the
> /64 for P-t-P, but set the few static ND entries you need, and turn
> off dynamic ND.  An out-of-band provisioning system could add static
> ND entries as needed.
>
> Another idea, perhaps more useful for client LANs, would be to have a
> fixed mapping between IPv6 IID and MAC address.  Use DHCPv6 to force

Chuck, you are certainly correct that if ND resolution can be
deactivated for an interface, there won't be an ND exhaustion problem
on it.  That's why I characterize the problem as ND exhaustion.  :-)
Whether or not this is practical for a given environment is up to the
operators to decide.

I, for example, know it is much easier for me to configure a /126
P-t-P than keep static ND entries and disable ND on those links.  In
my own backbone, your suggestion can be practical, but what about
customer links?  If the customer changes his device, he may present a
different MAC address to my interface.  Then I've got a static ND
entry pointing to his old MAC address... resulting in a ticket, and
ops work, which would not have been necessary with a simple /126.

DHCPv6 with snooping and learning disabled would be great for the
datacenter LAN if I thought I could get customers to bite off on it.
When vendors begin delivering this feature it is something we will
strongly consider.  I don't know if customers will prefer to have this
and need to run a DHCPv6 client, or prefer to have a /120 (or similar)
for the approximate number of addresses they plan to use.

I am not closed to alternatives.  I want to give my customers /64s as
soon as it becomes practical and production-ready.  That is why we
always reserve a /64 for each subnet, even though it is provisioned as
a longer subnet.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Writable SNMP

[Jeff Wheeler]

On Tue, Dec 6, 2011 at 11:07 AM, Keegan Holley
 wrote:
> For a few years now I been wondering why more networks do not use writable
> SNMP.  Most automation solutions actually script a login to the various

I've spent enough time writing code to deal with SNMP (our own stack,
not using Net-SNMP or friends) to have a more in-depth understanding
of SNMP's pitfalls than most people.  It is TERRIBLE and should be
totally gutted and replaced with something more sane, less likely to
have bugs, etc.  There is a good reason why many major bugs have
popped up over the years allowing devices to be crashed with crafted
SNMP packets -- it's honestly not that easy to get right, especially
if you really implement every possible encoding so some random
customer with a brain-damaged SNMP client stack won't come crying to
you that his client won't work.

Juniper does not support writing via SNMP.  I am glad.  Hopefully that
is the first step toward not supporting SNMP at all.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

OpenBGPd problems relating to misuse of RESERVED bits in BGP Attribute Flags field

[Jeff Wheeler]

I had two downstream BGP customers experience problem with an OpenBGPd bug
tonight.  Before diving into detail, I would like to link this mailing list
thread, because this is not a new issue and a patch is available:
http://www.mail-archive.com/misc@openbsd.org/msg115071.html

For the following DFZ routes, I see wrong use of the fifth bit in the
Attribute Flags field:
  Aggregator (7), length: 8, Flags [OT+8]:  AS #68, origin
192.65.95.253
0x:   0044 c041 5ffd
  Updated routes:
128.165.0.0/16
141.111.0.0/16
192.65.95.0/24
192.12.184.0/24
204.121.0.0/16

According to RFC 4271 page 17, "the low-order four bits of the Attribute
Flags octet are unused.  They MUST be zero when sent and MUST be ignored
when received."  I read "ignored" to mean, don't tear down the BGP session
and print a cryptic error that the user probably will be unable to debug.
 The OpenBGPd guys clearly agree and have supplied a patch, so affected
users should visit the above mailing list link, and install it.

Here are my notes for this RFC page and a small diagram of the packet
header, because surprisingly, there isn't one in the RFC already
http://inconcepts.biz/~jsw/img/1121129aa-rfc4271pg17scan.jpg  Sorry about
the poor quality of this, but it is past 3am here, and I know of several
operators (besides my downstream customers) who are experiencing this
problem right now.

If I were someone who is broken by this right now, I would either patch my
OpenBGPd or ask your eBGP neighbors not to send you the above five routes
(filtering it on your own OpenBGPd router probably won't help.)

Thanks, I hope this is helpful
-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: 32-bit ASes at routeviews

[Jeff Wheeler]

On Mon, Dec 17, 2012 at 6:14 AM, Claudio Jeker 
wrote:
> This can happen when a old 2-byte only routers are doing prepends with the
> neighbor address (4-byte). Then the magic in the 4-byte AS RFC to fix up
> ASPATH has no chance to work and you will see 23456.

After a careful re-read of RFC4893 section 4.2.3 Processing Received
Updates, I am fairly sure it is either an implementation issue with the
involved 4-octet ASN routers, or else their transit providers are using
as-path-*expand* when learning their routes for some reason (customers ask
for the strangest things.)

The specification for 4-octet AS refers to "old" and "new" BGP speakers,
which I'll do here:

When NEW speaker receives a route from an OLD speaker, its job is to make
AS_PATH and AS4_PATH the same length by using ASNs from from AS_PATH, which
cannot have been inserted into AS4_PATH by the OLD speaker(s) that do not
support the Attribute.

If a NEW speaker implements as-path-prepend incorrectly, and puts 23456
(AS_TRANS) into AS4_PATH instead of his real ASN, then the route passes
through some OLD speakers and out to a NEW one again, the second NEW
speaker has no opportunity to reconstruct the correct path.

On the other hand, if an OLD speaker is configured for as-path-*expand* as
it learns routes from a NEW speaker, then it may insert AS_TRANS into the
AS_PATH but no entries are being pushed to AS4_PATH.  This is a limitation
of the specification and cannot be avoided.  In effect, the use of as-path-*
expand* at a NEW->OLD boundary where the NEW router has a 4-octet ASN and
OLD router is performing *expand* means the correct AS_PATH cannot be
rebuilt.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Cloudflare is down

[Jeff Wheeler]

On Mon, Mar 4, 2013 at 9:51 AM, Leo Bicknell  wrote:
> will fix the problem.  It won't.  Next time the issue will be
> different, and the same undertrained person who missed the packet
> size this time will miss the next issue as well.  They should all be
> sitting around saying, "how can we hire compentent network admins for
> our NOC", but that would cost real money.

I think that is hard because virtually all training / education in our
industry is based on procedures, not on concepts.

Pick up any book about networking and you'll find examples of how to
configure a lab of Cisco 2900s so you can pass an exam.  Very few that
go into conceptual detail or troubleshooting of any kind.  Educational
programs suffer from the same flaw.

There are exceptions to this rule, but they are very few.  I'm sure
many NANOG readers are familiar with "Interdomain Multicast Routing,"
for example.  It is an excellent book because it covers concepts and
compares two popular vendor platforms on a variety of multicast
topics.

We have lots of stupid people in our industry because so few
understand "The Way Things Work."

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: Mitigating DNS amplification attacks

[Jeff Wheeler]

On Tue, Apr 30, 2013 at 8:35 PM, Jared Mauch  wrote:
> Please provide advice and insights as well as directing customers to the 
> openresolverproject.org website. We want to close these down, if you need an 
> accurate list of IPs in your ASN, please email me and I can give you very 
> accurate data.

I think that a public list of open-resolvers is probably overdue, and
the only way to get them fixed.

It is trivial to scan the entire IPv4 address space for DNS servers
that do no throttling even without the resources of a malicious
botnet.

Smurf was only "fixed" because, as there were fewer networks not
running `no ip directed-broadcast,` the remaining amplification
sources were flooded with huge amounts of malicious traffic.  The
public list of smurf amplifiers turned out to be the only way to
really deal with it.  I predict the same will be true with DNS.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

NANOG58 parking

[Jeff Wheeler]

I noticed that some folks were unhappy with the parking fee in Orlando.

The Roosevelt New Orleans, for NANOG 58, tells me that the only
on-site parking is valet for $42/day.  Anyone planning to drive or
stay at a different hotel may want to consider that in advance.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: De-bogon not possible via arin policy.

[Jeff Wheeler]

On Wed, Dec 14, 2011 at 4:15 PM, Cameron Byrne  wrote:
> Fyi, I just was rejected from arin for an ipv4 allocation. I demonstrated I
> own ~100k ipv4 addresses today.
>
> My customers use over 10 million bogon / squat space ip addresses today,
> and I have good attested data on that.

Cameron,

I have a client who went through the same problem in early-2011.  They
had several thousand residential and small business end-users behind
NAT and wished to provision public IP addresses for them.  They
assumed ARIN would be pleased to issue an appropriate block for their
project.  David Huberman rejected their request outright and told them
to request provider space, renumber the customers to provider IPs, and
then apply to ARIN again and renumber their network a second time.
The client did not bother to involve me until after they had already
been told to FOAD.

This is clearly a counter-productive waste of time, but if the client
had applied using the immediate need process, and provided the
additional supporting documentation required by that, I think they
would not have had this problem.  The analyst you worked with should
have suggested a different application procedure or otherwise worked
with you to facilitate your request.  Sometimes the ARIN staff are
nice and helpful, sometimes they are not.  It depends on who you get
assigned to, price of tea in china, phase of the moon, etc.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: local_preference for transit traffic?

[Jeff Wheeler]

On Thu, Dec 15, 2011 at 1:07 AM, Keegan Holley
 wrote:
> Had in interesting conversation with a transit AS on behalf of a customer
> where I found out they are using communities to raise the local preference

That sounds like a disreputable practice.

While not quite as obvious, some large transit ASes, like Level3,
reset the origin to I (best) sometime between when they learn it and
when they announce it to their customers and peers.  This similarly
causes them to suck in a bit more traffic than they might otherwise.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: local_preference for transit traffic?

[Jeff Wheeler]

On Thu, Dec 15, 2011 at 2:24 AM, Keegan Holley
 wrote:
> I always assumed that taking in more traffic was a bad thing.  I've heard
> about one sided peering agreements where one side is sending more traffic
> than the other needs them to transport. Am I missing something?  Would this
> cause a shift in their favor allowing them to offload more customer traffic
> to their peers without complaint?

Well, if Level3 wanted less ingress traffic, they would probably stop
this practice.  I would imagine they thought about it carefully.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: De-bogon not possible via arin policy.

[Jeff Wheeler]

On Thu, Dec 15, 2011 at 4:54 PM, Joel jaeggli  wrote:
> We know rather alot about the original posters' business, it has ~34
> million wireless subscribers in north america. I think it's safe to
> assume that adequate docuementation could be provided.

I missed the post where he supplied this information.  I guess his
company should have cheated the system, with full complicity of ARIN,
like Verizon Wireless did a few years ago.
http://marc.info/?l=nanog&m=123406577704970&w=4

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: IPv6 RA vs DHCPv6 - The chosen one?

[Jeff Wheeler]

On Fri, Dec 23, 2011 at 4:13 PM, Mohacsi Janos  wrote:
> If you can limit number of ARP/NDP entries per interfaces and you complement
> RAGuard and DHCPv4 snooping your are done.

That depends on how ARP/ND gleaning works on the box.  In short, Cisco
already has a knob to limit the number of ND entries per interface on
some of their kit, and it is not a solution, only a damage mitigation
measure.  http://inconcepts.biz/~jsw/IPv6_NDP_Exhaustion.pdf

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts

Re: subnet prefix length > 64 breaks IPv6?

[Jeff Wheeler]

On Wed, Dec 28, 2011 at 10:19 AM, Ray Soucy  wrote:
> There are a few solutions that vendors will hopefully look into.  One
> being to implement neighbor discovery in hardware (at which point
> table exhaustion also becomes a legitimate concern, so the logic
> should be such that known associations are not discarded in favor of
> unknown associations).

Even if that is done you are still exposed to attacks -- imagine if a
downstream machine that is under customer control (not yours) has a
whole /64 nailed up on its Ethernet interface, and happily responds to
ND solicits for every address.  Your hardware table will fill up and
then your network has failed -- which way it fails depends on the
table eviction behavior.

Perhaps this is not covered very well in my slides.  There are design
limits here that cannot be overcome by any current or foreseen
technology.  This is not only about what is broken about current
routers but what will always be broken about them, in the absence of
clever work-arounds like limits on the number of ND entries allowed
per physical customer port, etc.

We really need DHCPv6 snooping and ND disabled for campus access
networks, for example.  Otherwise you could give out addresses from a
limited range in each subnet and use an ACL (like Owen DeLong suggests
for hosting environments -- effectively turning the /64 into a /120
anyway) but this is IMO much worse than just not configuring a /64.

On Wed, Dec 28, 2011 at 10:45 AM,   wrote:
> I'm afraid I don't believe this is going to happen unless neighbor
> discovery based attacks become a serious problem. And even then it would
> take a long time.

The vendors seem to range from "huh?" to "what is everyone else
doing?" to Cisco (the only vendor to make any forward progress at all
on this issue.)  I think that will change as this topic is discussed
more and more on public mailing lists, and as things like DHCPv6
snooping, and good behavior when ND is disabled on a subnet/interface,
begin to make their way into RFPs.

As it stands right now, if you want to disable the IPv6 functionality
(and maybe IPv4 too if dual-stacked) of almost any datacenter /
hosting company offering v6, it is trivial to do that.  The same is
true of every IXP with a v6 subnet.  I think once some bad guys figure
this out, they will do us a favor and DoS some important things like
IXPs, or a highly-visible ISP, and give the vendors a kick in the
pants -- along with operators who still have the "/64 or bust"
mentality, since they will then see things busting due to trivial
attacks.

-- 
Jeff S Wheeler 
Sr Network Operator  /  Innovative Network Concepts