Scott,
On Oct 14, 2008, at 9:08 AM, Scott Doty wrote:
First, the good news: so far, the NANOG conference has been very
valuable and
content-rich, covering a lot of issues that need to be discussed.
For that, I am grateful.
Thank you. We worked hard to make it valuable.
But now, the bad news(?): Maybe it's just me & my paranoia, but do
I detect
an inkling of "murk spam" going on with some presentations?
Not sure what you mean by "murk spam". Thats a term that died years
ago. And it really related to people claiming that spam was "in
compliance with federal laws". But I think I can guess your intentions
from the tone of your email, so let me try and respond.
Because there seems to be a fundamental misunderstanding, either on
my part,
or the part of certain vendors: I'm hear to discuss ideas & freely
share
them, and they are here to discuss (it would seem) their products.
Sometimes
both goals coincide, and that is fine...but...
When a vendor at the security BOF starts showing documents that are
"company
confidential", and trying to whip up a climate of fear, that we
should all
deploy their product in front of our recursive name servers, i get
this
funny feeling that I am being "murk spammed".
Well, that's interesting. I see your last NANOG was 9, in February of
1997. So "Welcome back!". We're glad to have you here in person.
Things have changed slightly since then. NSP-SEC never existed in
1997. It really came about in the early 2000's where it was developed
as a forum for actual operators to share views and thoughts, generally
in real time, to help the 'net in general survive disruption,
malicious or otherwise. It has really worked pretty well, so if you
qualify, I'd encourage you to get involved. See http://puck.nether.net/mailman/listinfo/nsp-security
for info.
The NSP-SEC bof at NANOG is not quite the same environment as the NSP-
SEC mailing list, but it generally includes the same people, plus
others from the operations community who take the effort to attend
NANOG, and so are sort of self-selected as being "one of the
operators" with an already working amount of clue about the subjects
that are being discussed. Additionally, the concept of a "trusted
environment" still sorta applies. You may not have realized it, but
unlike all other sessions at NANOG, the slides are not published, they
are not available online, and the session is not broadcast. So
"Confidential" was there to remind folks in the BoF that this was a
non-public (for a skewed version of public) presentation.
Having explained that bit of history which gives you a general
background, let me deal with some specifics.
Perhaps that is my own perspective (& paranoia?), but I found the CERT
gentleman's call to monitor icmp backscatter on our authoritative
nameservers far more informative -- and open.
I don't think anyone from CERT presented. Perhaps you meant Barry
Green from Juniper's CERT team? Another "vendor"? Well, as you'll see
further on, not really. In this context, like everyone else who
presented, he was there as an operator, sharing knowledge and
experience. But I digress...
But I was disappointed with two vendors and their presentations: the
first
had the tactic of saying "DNSSEC is the actual solution" when asked
about
why their product would be necessary...completely ignoring the fact
that
their proprietary "interim solution" was by no means the only way to
prevent
cache poisoning attacks. Indeed, I would daresay it isn't the best,
either
by a BCP perspective, or a cost analysis perspective.
While we may disagree on your last claim (and I actually have a few
years of experience to help me argue my point), I specifically said
there were a) solutions that solved part of the problem (switching to
TCP, detecting and blocking cache poisoning attacks) and b) the right
solutions like DLV and DNSSEC that will take some time to be deployed.
And I then made sure everyone heard me when I said that we need to
find an interim solution that can be deployed *now*, until DNSSEC
exists in a useful footprint. I ignore *nothing*. If you have another
solution that solves the same problems that has running code now,
please share it with all of us. Remember, it has to scale, it has to
solve all of the problems, and it has to be implementable across a
range of levels of clue.
To put a finer point on this, i should say that i found myself
discomforted
by a presentation suggesting that I should put their proprietary
appliances
between my recursive name servers & the Net, and I am grateful that
Mr.
Vixie stood up and said that there are other ways of dealing with the
problem.
Indeed. Read further.
Fortunately, said vendor had a table at "beer and gear", so I was
able to
talk with one of their representatives -- and learned that they have
just as
much trouble with automatic detection of attacks designed to look
like a
"slashdotting"...which cleared up the mystery as to why it wasn't on
the
graphs.
Because this is a real problem: anybody, with sufficient knowledge &
preparation can vandalize _anybody's_ network. Showing me a graph
that ping
floods happen all the time doesn't impress me -- what would impress
me is
going over the actual methods, algorithms (and heuristics?) used in
these
attack mitigation appliances.
Because, the "best" attack mitigation appliance vendor would seem to
have
100% of their market, and thus, charge exhorbant prices for their
product(s). When I brought this up with Mr. Vendor, his first
reaction was
to point out that the cost was less than a home-grown solution.
When I
raised the question of open source software to do the same thing, his
reaction was to ask: "oh? who's going to write it?"
And that right there would seem to be a bit of bravado, perhaps
fueled by a
misunderstanding of the role that FOSS has played on the Net.
Fortunately -- and again, I am grateful for this -- the ISC was
represented
in the security BOF, presenting the SIE concept...as well as what
applications _already exist_ to detect and mitigate various
attacks. One
demonstration that blew me away: detecting a botnet being set up
for a
phishing attack...and preventing the attack before it even started.
Cool. I'm glad you saw value from that "vendor".
Seriously. SIE is good stuff.
So in conclusion, I'll say this: the last NANOG I attended was
NANOG 9 --
and i remember that being a more challenging environment for vendors.
Probably the biggest problem discussed back then was head-of-line
blocking
on a vendor's switches. _That_ is the kind of content that i have
found
valuable, both on this list, and at a conference.
And so: If I weren't so knock-kneed in public venues,
I would probably be doing what i would like to call on conference
participants to do: if someone gives a presentation that includes
their own
proprietary black-box "solution", I think the best benefit for NANOG
would
be to point out alternatives.
*I* was the "vendor" at the security BOF you took aim at. Except I am
not a vendor in this environment. I am an operator. Just like ISC
(Vixie) and McPherson (Arbor) and Greene (Juniper) etc. We are there
as operators and *none* of us was selling *anything. We were
describing issues that we currently are facing as operators, and
solutions we have developed. You're not alone amongst "newcomers" in
missing the point, so don't be hard on yourself ;-). In my case,
*nothing* was being sold, other than *a* solution, which I am actually
*giving* away to networks that matter in solving the probelm, and
picking up the costs myself. I assume you missed that. And the reason
I was doing that with a *proprietary* solution was because the open
source solution is *not yet ready* for prime time, mainly because it
(they) have not solved the wide implementation challenge. And *we*
need to find a solution today while the open source (and best
solution) gets rolled out effectively. Paul (also a "vendor" in the
same vein, but an operator in the BoF forum) answered the question of
whether there was another solution by saying "there is in Bind 9.6" -
his product, which was released a couple of weeks ago.
I referred to it in my presentation, as a solution, along with
DNSSEC. It's called DLV. Unfortunately, and Paul admits it, there are
challenges to widespread adoption. It works, but there is no business
case that makes it easy to roll out. And therein lies the challenge.
My customers need it today. And if it isn't out there in wide use, *it
doesn't solve the problem*. So I am solving that by picking up the tab
myself, and being reimbursed by the people I am a vendor to, my
customers. And they're happy to pay for it. None of them were at the
bof. Well, not strictly true, but not in numbers to matter. But
hopefully you get the point. And you now understand that in the BoF we
are all working to try and *solve* problems, not sell products. I'm
sorry you failed to grok that difference.
Finally, despite your knocking knees, you should have stood up and
questioned anything you heard, or misunderstood. Then you would have
had a better experience of the bof. As a member of the Program
Committee and coincidentally the host of this NANOG, I'm sorry we
didn't do a better job. We're trying to get better. I think that this
was one of the best NANOGs we've ever had. But I'm biased, especially
this time ;-).
As an aside, since you were last at a NANOG, we now have Beer 'n Gear,
where Vendors have the opportunity to show off their wares, and in
exchange they support and underwrite some of the costs of what is a
pretty slick conference. I'm not sure why you believe that the vendor
pitching his/her products at Beer 'n Gear is in some way violating the
sacred rule against talking about a product. The B&G specifically
provides the controlled environment and tradeoff. And *most* operators
appreciate it, and make really good use of the opportunity to learn
about new products that actually matter in such a useful environment.
In one place we get to talk to actual engineers, about their products,
together with 500 fellow operators who ask questions we may not even
know we should ask.
If you have any other questions about my presentation, or the program,
please feel free to ask directly.
-Scott
p.s. sorry for the long post.
Ditto for the response. But I have to assume you were not the only one
who may have missed key points. Thanks for coming back. Hopefully
we'll see you in the Dominican Republic next January.
