Re: Filter NTP traffic by packet size?

[Keegan Holley]

On Feb 25, 2014, at 12:22 PM, Staudinger, Malcolm 
 wrote:

> Why wouldn't you just block chargen entirely? Is it actually still being used 
> these days for anything legitimate?
> 

More politely stated, it’s not the responsibility of the operator to decide 
what belongs on the network and what doesn’t.  Users can run any services 
that’s not illegal or even reuse ports for other applications.  That being said 
commonly exploited ports (TCP 25 for example) are often blocked.  This is 
usually done to block or protect an application though not to single out a 
particular port number.

> Malcolm Staudinger
> Information Security Analyst | EIS
> EarthLink
> 
> E: mstaudin...@corp.earthlink.com
> 
> -Original Message-
> From: Blake Hudson [mailto:bl...@ispn.net] 
> Sent: Tuesday, February 25, 2014 8:58 AM
> To: nanog@nanog.org
> Subject: Re: Filter NTP traffic by packet size?
> 
> I talked to one of our upstream IP transit providers and was able to 
> negotiate individual policing levels on NTP, DNS, SNMP, and Chargen by UDP 
> port within our aggregate policer. As mentioned, the legitimate traffic 
> levels of these services are near 0. We gave each service many times the 
> amount to satisfy subscribers, but not enough to overwhelm network links 
> during an attack.
> 
> --Blake
> 
> Chris Laffin wrote the following on 2/23/2014 8:58 AM:
>> Ive talked to some major peering exchanges and they refuse to take any 
>> action. Possibly if the requests come from many peering participants it will 
>> be taken more seriously?
>> 
>>> On Feb 22, 2014, at 19:23, "Peter Phaal"  wrote:
>>> 
>>> Brocade demonstrated how peering exchanges can selectively filter 
>>> large NTP reflection flows using the sFlow monitoring and hybrid port 
>>> OpenFlow capabilities of their MLXe switches at last week's Network 
>>> Field Day event.
>>> 
>>> http://blog.sflow.com/2014/02/nfd7-real-time-sdn-and-nfv-analytics_19
>>> 86.html
>>> 
 On Sat, Feb 22, 2014 at 4:43 PM, Chris Laffin  wrote:
 Has anyone talked about policing ntp everywhere. Normal traffic levels are 
 extremely low but the ddos traffic is very high. It would be really cool 
 if peering exchanges could police ntp on their connected members.
 
> On Feb 22, 2014, at 8:05, "Paul Ferguson"  
> wrote:
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
>>> On 2/22/2014 7:06 AM, Nick Hilliard wrote:
>>> 
>>> On 22/02/2014 09:07, Cb B wrote:
>>> Summary IETF response:  The problem i described is already solved 
>>> by bcp38, nothing to see here, carry on with UDP
>> udp is here to stay.  Denying this is no more useful than trying 
>> to push the tide back with a teaspoon.
> Yes, udp is here to stay, and I quote Randy Bush on this, "I 
> encourage my competitors to block udp."  :-p
> 
> - - ferg
> 
> 
> - --
> Paul Ferguson
> VP Threat Intelligence, IID
> PGP Public Key ID: 0x54DC85B2
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2.0.22 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iF4EAREIAAYFAlMIynoACgkQKJasdVTchbJsqQD/ZVz5vYaIAEv/z2kbU6kEM+KS
> OQx2XcSkU7r02wNDytoBANVkgZQalF40vhQED+6KyKv7xL1VfxQg1W8T4drh+6/M
> =FTxg
> -END PGP SIGNATURE-
> 
> 

Re: Managing IOS Configuration Snippets

[Keegan Holley]

Putting aside the fact that snippets aren’t a good way to conceptualize 
deployed router code, my gut still tells me to question the question here.  The 
first is does this stuff change often enough to warrant a fancy versioning 
solution?  I have yet to see NTP deployed in a different way than when I first 
learned to configure it.  Next, when it does change how often is it not rolled 
out to every router.  If NTP or CPP or SNMP or some other administrative option 
were configured differently across my network I would want to audit it and fix 
not version control.  What if some of the configs don’t match the defined 
versions?  It may be better to create standard templates and version them in 
SVN or GIT and then use config backups to track which devices have the standard 
configs.  There are some for pay tools that can search for certain statements 
on various boxes and either alert or remediate when differences are found. 


On Feb 26, 2014, at 4:22 PM, Ryan Shea  wrote:

> Howdy network operator cognoscenti,
> 
> I'd love to hear your creative and workable solutions for a way to track
> in-line the configuration revisions you have on your cisco-like devices.
> Let me clearify/frame:
> 
> You have a set of tested/approved configurations for your routers which use
> IOS style configuration. These configurations of course are always refined
> and updated. You break these pieces of configuration into logical sections,
> for example a configuration file for NTP configuration, a file for control
> plane filter and store these in some revision control system. Put aside for
> the moment whether this is a reasonable way to comprehend deployed
> configurations. What methods do some of you use to know which version of a
> configuration you have deployed to a given router for auditing and update
> purposes? Remarks are a convenient way to do this for ACLs - but I don't
> have similar mechanics for top level configurations. About a decade ago I
> thought I'd be super clever and encode versioning information into the snmp
> location - but that is just awful and there is a much better way everyone
> is using, right? Flexible commenting on other vendors/platforms make this a
> bit easier.
> 
> Assume that this version encoding perfectly captures what is on the router
> and that no person is monkeying with the config... version 77 of the
> control plane filter is the same everywhere.

Re: Filter NTP traffic by packet size?

[Keegan Holley]

On Feb 26, 2014, at 12:44 PM, Brandon Galbraith  
wrote:

> On Wed, Feb 26, 2014 at 6:56 AM, Keegan Holley  wrote:
> > More politely stated, it’s not the responsibility of the operator to decide 
> > what belongs on the network and what doesn’t.  Users can run any services 
> > that’s not illegal or even reuse ports for other applications.  That being 
> > said commonly exploited ports (TCP 25 for example) are often blocked.  This 
> > is usually done to block or protect an application though not to single out 
> > a particular port number.
> 
> Don't most residential ISPs already block port 25 outbound? 
> http://www.postcastserver.com/help/Port_25_Blocking.aspx
> 
> Blocking chargen at the edge doesn't seem to be outside of the realm of 
> possibilities.

As I said, SMTP is blocked because it’s the default port for a commonly run and 
often misconfigured application.  Blocking the chargen port is definitely 
reasonable, but it’s not a popular application.  Most people use the port as an 
clever non-default port for some other service like ssh.

Re: Managing ACL exceptions (was Re: Filter NTP traffic by packet size?)

[Keegan Holley]

It depends on how many customers you have and what sort of contract you have 
with them if any.  A significant amount of attack traffic comes from 
residential networks where a “one-size-fits-all” policy is definitely best.

On Feb 26, 2014, at 4:01 PM, Jay Ashworth  wrote:

> - Original Message -
>> From: "Brandon Galbraith" 
> 
>> On Wed, Feb 26, 2014 at 6:56 AM, Keegan Holley 
>> wrote:
>>> More politely stated, it’s not the responsibility of the operator to
>>> decide what belongs on the network and what doesn’t. Users can run any
>>> services that’s not illegal or even reuse ports for other
>>> applications.
> 
>> Blocking chargen at the edge doesn't seem to be outside of the realm
>> of possibilities.
> 
> All of these conversations are variants of "how easy is it to set up a
> default ACL for loops, and then manage exceptions to it?".
> 
> Assuming your gear permits it, I don't personally see all that much 
> Bad Actorliness in setting a relatively tight bidirectional ACL for
> Random Edge Customers, and opening up -- either specific ports, or
> just "to a less-/un-filtered ACL" on specific request.
> 
> The question is -- as it is with BCP38 -- *can the edge gear handle it*?
> 
> And if not: why not?  (Protip: because buyers of that gear aren't 
> agitating for it)
> 
> Cheers,
> -- jra
> -- 
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274
> 

Re: Managing IOS Configuration Snippets

[Keegan Holley]

On Feb 28, 2014, at 9:11 AM, Ryan Shea  wrote:

> Keegan, don't get me wrong, I am not suggesting that even if version numbers 
> were happily encoded in robust comments that this would be the same as 
> actually digesting the configuration. If the function of checking using 
> 'fancy versioning' is not an operational best practice, what do you suggest 
> (all-knowing/singing/dancing tool which understands the configuration and 
> your intent aside)? You say IF NTP or CPP were configured differently - with 
> a large enough network there will always be configurations which have 
> differences. With that as an operational constant, how do you determine which 
> devices have the latest iteration of your line vty configuration.

That’s what I mean.  The things that lend well to to version tracking don’t 
tend to change much.  How many ways are there to configure VTY lines, or NTP, 
or CPP, or even OSPF and if we’re talking about an access ACL why not just 
audit the configs to make sure that all the entries are there.  Am I really 
going to care that one router has version 1.0 versus another router that has 
version 2.2.12 build9?  It’s not source code..  
> 
> How often will a change not be rolled out to every router. This is again 
> related to the size and churn of the network, but my practical estimation is 
> that once you get into thousands of routers there will almost always be some 
> that get missed.

Again, a router that was missed is a reason for audit and remediation not 
versioning.  If you find a router with config missing does it really matter 
what version it’s on and when that version was valid?  Not in my experience.

> Comprehensive auditing is very important, and arguably more useful than 
> version checking - but it requires that you make knowledgeable and complete 
> assertions. I assert the my snmp config should look like the snmp snippet 
> version 77 is easier to grok than "make sure our community string is not set 
> to public" (and repeat hand-crafted audit logic for every segment of the 
> config).

There may be some differences, but those are normally due to equipment 
lifecycle, mergers/consolidations and such.  It’s easy to refer to something as 
the config for a particular platform or company than a version number.  This 
can be tracked in GIT or SVN.  Even then there will not be constant changes.  
I’d lean towards standardization.  So the equipment that cannot adhere to the 
defined standards probably won’t evolve much on it’s own.
> 
> What if some of the configs don't match the defined versions? This is why it 
> may make sense to break snippets into functional areas. "Just fix it" might 
> be sane for a banner, but squashing an interface mtu change that was put 
> there for a reason could end in tears. I consider this bit out of the scope 
> of the question, but yes it is another important problem.

I wasn’t saying just fix it.  I was saying that router configs don’t lend well 
to versioning.  With software for example, if something is different it might 
be a different version of that application with compatibility issues, 
dependencies, library issues, etc.  When it’s a router config chances are 
someone fat-fingered something.  Most of the time the best thing to do is to 
fix or at least alert on the error, not to record it as a valid config version. 
 Again, this is for things that lend themselves to snippets.  ACL’s, NTP, SNMP, 
CPP, even Spanning-tree.  Not for things like interface IP’s or static routes 
that may be different across different boxes or location.  If you’re referring 
to the latter I may have misunderstood your question..

> 
> 
> On Thu, Feb 27, 2014 at 10:03 PM, Christopher Morrow 
>  wrote:
> On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley  wrote:
> > Putting aside the fact that snippets aren't a good way to conceptualize 
> > deployed router code, my gut still tells me to question the question here.  
> > The first is does this stuff change often enough to warrant a fancy 
> > versioning solution?  I have yet to see NTP deployed in a different way 
> > than when I first learned to configure it.  Next, when it does change how 
> > often is it not rolled out to every router.  If NTP or CPP or SNMP or some 
> > other administrative option were configured differently across my
> 
> sure, so you're saying that a large bit (maybe) of the router config
> is 'one size fits all' and 'never changes' where 'never' is really
> 'very infrequently'.
> 
> sure, agreed... but there are parts of the config that do change more
> frequently (depending on the network perhaps)... how do you go about
> seeing which version / setup is deployed EXCEPT by building a
> home-grown 'config parser' and seeing that &#x

Re: Managing ACL exceptions (was Re: Filter NTP traffic by packet size?)

[Keegan Holley]

+1 in my experience uRPF get’s enabled, breaks something or causes confusion 
(usually related to multi-homing) and then get’s disabled.

On Feb 28, 2014, at 11:49 AM, Christopher Morrow  
wrote:

> On Fri, Feb 28, 2014 at 9:02 AM, Ray Soucy  wrote:
>> If you have uRPF enabled on all your access routers then you can
>> configure routing policy such that advertising a route for a specific
>> host system will trigger uRPF to drop the traffic at the first hop, in
>> hardware.
> 
> note that 'in hardware' is dependent upon the model used...
> note that stuffing 2k (or 5 or 10 or...) extra routes into your edge
> device could make it super unhappy.
> 
> your points are valid for your designed network... they may not work 
> everywhere.
> making the features you point out work better or be more widely known
> seems like a great idea though :)

Re: Managing IOS Configuration Snippets

[Keegan Holley]

On Feb 28, 2014, at 9:35 PM, Dale W. Carder  wrote:

> Thus spake Keegan Holley (no.s...@comcast.net) on Fri, Feb 28, 2014 at 
> 09:49:19AM -0500:
>> I wasn’t saying just fix it.  I was saying that router configs don’t lend 
>> well to versioning.  
> 
> Um, what?
> 
> $> rlog r-cssc-b280c-1-core.conf | grep 'total revision'
> total revisions: 2009;selected revisions: 2009

I wish you were here to see my eyes rolling.. 2009 versions of something are no 
more grok-able than one current version.  Congrats, you have a config backup 
system.
> 
>> When it’s a router config chances are someone fat-fingered something.  Most 
>> of the time the best thing to do is to fix or at least alert on the error, 
>> not to record it as a valid config version. 
> 
> We have our operators manually check in revisions (think in rcs terms:
> co -l router, go do work, verify it, ci -u router) rather than
> unsolicited / cron-triggered checkins.  Then the check-in message
> contains the operator's description text of the change and often a
> ticket number.  So there slightly fewer fat-finger configs checked in.

That’s not what the OP was looking for AFAIK.  This is just change management.

> 
> Dale

Re: Any experience with Comcast digital voice for OOB (offlist is fine)

[Keegan Holley]

As others have said modems require POTS or at least a PBX line.  Also isn’t the 
hand-off fog VoIP ethernet?  You wouldn’t be able to stick that into the RJ-11 
port in the modem.  It would be easier to use the comcast internet connection 
with some sort of IPsec tunnel for OOB.  It’s cheap and mostly reliable.

If you’re looking for a better solution see the thread on OOB gear RE: 
opengear.  They are multi-port and support, POTS, wifi and 3G for access.

On Feb 28, 2014, at 2:27 PM, eric-l...@truenet.com wrote:

> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> F: 610-429-3222
> 
> 
> 
> 
> 

Re: How to catch a cracker in the US?

[Keegan Holley]

I’ve seen past employers contact the FBI for a similar issue, but we had 
control of the network and logs in question so that made it easier.  You may be 
able to contact interpol or a similar agency in the EU.  They will at least be 
able to tell you the right agency to call.

You can also have a lawyer contact comcast on your behalf.  Many times a 
company will retrieve and store logs in preparation of various legal 
proceedings.  Comcast is a very large company so there’s no way to be sure that 
this will spur them into action, but it’s a start.

On Mar 11, 2014, at 3:00 AM, Markus  wrote:

> Hi,
> 
> I'm an ISP in Germany and a cracker (not a hacker :) ) has targeted a 
> customers of mine in the last days. The cracker was successful and caused 
> financial damage / was successful with data theft. I set a trap and finally 
> caught his real IP address - a Comcast user in the US (100% not a proxy or 
> bot). What would be the next steps to pursuit him? If I contact local 
> authorities here in Germany I'm afraid months will pass by and Comcast will 
> have possible already deleted their logs by then (?). Any advice?
> 
> Thank you!
> Markus
> 

Re: Level 3 blames Internet slowdowns on Technica

[Keegan Holley]

On Mar 21, 2014, at 12:13 PM, Naslund, Steve  wrote:

> We don't know because the service provider rolls that cost up along with the 
> services they sell.  That is my point.  They are able to spread the costs out 
> based on the profitable services they sell.  If they were not able to sell us 
> services I am not sure they could afford to provide that infrastructure.

Monthly fees do much more than finance the cost of infrastructure.  Most large 
providers take a significant margin.  It’s all about how these services are 
perceived.  The preservation of this margin is the number one reason why 
internet access isn’t considered a utility or a basic right today.  It also 
allows prices to increase unchecked, based on nothing other than the goals of 
specific companies. 

There are many places where infrastructure is subsidized and controlled by the 
government.  To them some things are more important than the need to make 
markets.  They just trust that the overall benefit to society is worth 
modifying the market.

>  In fact, having been a service provider I can tell you that I paid the LEC 
> about $4 a month for a copper pair to your house to sell DSL service at 
> around ten times that cost.  I am sure the LEC was not making money at the $4 
> a month and I know I could not fund a build out for that price.

Being a LEC is more profitable than anything else because they control the 
prices.  If you found an iLEC that charged $4 for something worth $40 that 
doesn’t mean being a LEC isn’t profitable.  After the long-distance carriers 
were forced to divest from local the LEC’s grew and quickly bought them.  It’s 
more profitable to be a LEC than to resell their services.  The only large 
carriers left are former LEC’s AKA baby bells.
> 

> -Original Message-
> From: Jim Popovitch [mailto:jim...@gmail.com] 
> Sent: Friday, March 21, 2014 11:07 AM
> To: Naslund, Steve
> Cc: Sholes, Joshua; Larry Sheldon; nanog@nanog.org
> Subject: Re: Level 3 blames Internet slowdowns on Technica
> 
> On Fri, Mar 21, 2014 at 11:48 AM, Naslund, Steve  wrote:
>> What do you mean by average monthly bill?
> 
> What is the average monthly (non-subsidized) access cost that your friends 
> and family pay each month?
> 
> -Jim P.
> 

Re: Level 3 blames Internet slowdowns on Technica

[Keegan Holley]

How come no one ever asks if competition is required?

On Mar 20, 2014, at 11:47 PM, David Miller  wrote:

> Unless I am reading the tea leaves wrong "competition" will require 
> "regulation".
> 
> 
> 
>  Original message 
> From: "Mike."  
> Date: 03/20/2014  21:56  (GMT-05:00) 
> To: nanog@nanog.org 
> Subject: Re: Level 3 blames Internet slowdowns on 
> 
>  Technica 
> 
> On 3/20/2014 at 4:17 PM Bryan Fields wrote:
> 
> |On 3/20/14, 12:34 PM, Blake Hudson wrote:
> |> The solution seems to be competition or regulation.
> |I'd prefer competition to regulation.  
> =
> 
> If real and true competition exists, yes.
> 
> 
> 
> 

Re: competition (was: Level 3 blames Internet slowdowns on Technica)

[Keegan Holley]

On Mar 21, 2014, at 2:21 PM, Jared Mauch  wrote:

> 
> On Mar 21, 2014, at 2:08 PM, Keegan Holley  wrote:
> 
>> How come no one ever asks if competition is required?
> 
> I think the issue here is there is competition, but those you are seen as 
> competing with are in a different strata providing the same service.

My question is competition and the market the goal at all?
> 
> eg: Cellular data competes with DSL/DOCSIS/FTT*
> 
> Now, due to speed, caps, etc.. it may not be a "fair" comparison, but this 
> isn't about fair, it's about "is there competition in the market".  I know 
> many folks that live outside the wired high-speed boundaries and things are 
> not getting any better there.  Most use some hotspot or similar for their 
> home connectivity.
> 
> Is there a market for high speed there?  certainly, but it's being filled by 
> other technology.  

Again why is the market so important?  The inhabitants of this list operate 
(some help develop) the most complex system created by our species to date.  It 
is one of the few truly global systems and brought with it a new era in human 
development.  We now have more information at our fingertips than at any point 
in history.  What do we argue about?  How to profit from it?  I’m not saying 
that profit is bad. I’m arrogant but not arrogant enough to think I can answer 
such a question.  It just fascinates me that no one questions it.

If an area isn’t considered not to be profitable it’s just fine that the 
internet doesn’t stretch there.  We don’t even have a definition of what 
profitable means.  It’s completely up to the ISP’s.  Still, businesses in that 
area are limited, children don’t do as well in school and in turn don’t have as 
much opportunity.  All of this happens, unquestioned in the name of profits.

> 
> There are many folks that work around these issues with other solutions, 
> including satellite, fixed wireless and/or microwave or even localized fiber 
> build-outs.  Look at the RUS/NTIA/BTOP focus, it was on getting the anchor 
> institutions well connected to provide a sense of community.  The challenge 
> is not everyone is equally equipped.  Merit (in my area) has fiber close to 
> me, but they don't offer services to anyone but existing members and have no 
> consumer offerings.
> 
> Market segmentation happens for a variety of reasons, sometimes economic, 
> sometimes complete differences in ROI models.

Market segmentation doesn’t happen as much as market consolidation.  There are 
now three (with a 4th that is close) major carriers in the US with enough 
market share to compete with each other.  There isn’t much segmentation because 
segmentation isn’t as profitable.

> 
> Nobody can afford to run universal fiber everywhere as a greenfield build, 
> but there are localized markets where it can make sense.  

That’s totally untrue.  What is affordable to a multi-billion dollar ISP 
anyway?  Are you saying they’d go bankrupt if they ran fiber everywhere?  No, 
it’s just that the infrastructure isn’t profitable in the short term.  There’s 
a reason why energy companies can’t make the same decision.

> Certainly it can make sense to connect some islands to each other via some 
> other technology.  Taking list prices from providers webpages, what cogent 
> used to list $4/meg, so that means (assuming everything is perfect) offering 
> 10Mb/s service at a home could possibly cost $40/mo for a provider, not 
> counting capital costs and other elements (support, customer acquisition 
> costs, bad debt, etc).
> 
> I'm sure folks can build networks for low cost, you can get a 1G 
> active-ethernet NID for sub-$150 with optics, but you still need to aggregate 
> and account for it somewhere.

Again why does everything have to move at the speed of profit?  At least here 
in the US anything that could remotely benefit society is always first shot 
through the prism of profit and the so-called free market.  Is a market with 
three major players and a 9-figure entry cost really free though?


> 
> - Jared

Re: What's a reasonable attack surface? (was: Re: wet-behind-the-ears whippersnapper yada yada)

[Keegan Holley]

I think people tend to go overboard in the planning phases for something
like this.  I remember rumors of a certain large ISP getting along fine for
several years installing routers with a password like "getsmein".  There are
plenty of groups that publish guidelines on ISP configuration as well as a
wealth of books on the subject.  You'll just have to start out with the
basics and learn as you go.  I'm not sure you can get a magic bullet from a
mailing list though.

2011/9/20 Jay Ashworth 

> - Original Message -
> > From: "Valdis Kletnieks" 
>
> > As long as there is *A* failure mode? Hmm.  mode
> > involving crazed ninjas with katanas loose in a switch room at one
> provider>.
> > Yep, it's unlikely crazed ninjas will attack the switch rooms at both
> > providers.
>
> I too am a Schneier fan.  But terrorists watch movies, too.
>
> Cheers,
> -- jr 'Once is happenstance...' a
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land Rover
> DII
> St Petersburg FL USA  http://photo.imageinc.us +1 727 647
> 1274
>
>
>

Re: Authoritative DNS server for 12.54.94.0/23 PTR

[Keegan Holley]

it looks like ATT still answers the queries.  I'd assume that any changes
would have to be authorized by the customer though.  Why not just call
Siemens Medical?

; <<>> DiG 9.6.0-APPLE-P2 <<>> -x 12.54.91.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21619
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;1.91.54.12.in-addr.arpa.INPTR

;; AUTHORITY SECTION:
54.12.in-addr.arpa.900INSOAxbru.br.ns.els-gms.att.net.
rm-hostmaster.ems.att.com. 1179 86400 1 60 172800



2011/9/27 Mike Gatti 

> Hello Nanog Members,
>
> We have been having some issue doing reverse lookups for ip's in the
> 12.54.94.0/23 prefix. We know that this block is assigned to AT&T and AT&T
> has assigned that block to Siemens Medical (based on whois queries). We are
> now trying to find out who would be the authoritative DNS server that would
> resolve PTR queries for these IP addresses. Would someone from AT&T (or
> Siemens) or someone that has that info please contact me offline to discuss?
>
> Thanks Everyone,
> --
> Michael Gatti
> cell.949.735.5612
> ekim.it...@gmail.com
> (UTC-8)
>
>
>
>
>
>

Re: Authoritative DNS server for 12.54.94.0/23 PTR

[Keegan Holley]

Looks like I typo'd the third octet.

2011/9/27 Keegan Holley 

> it looks like ATT still answers the queries.  I'd assume that any changes
> would have to be authorized by the customer though.  Why not just call
> Siemens Medical?
>
> ; <<>> DiG 9.6.0-APPLE-P2 <<>> -x 12.54.91.1
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21619
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;1.91.54.12.in-addr.arpa.INPTR
>
> ;; AUTHORITY SECTION:
> 54.12.in-addr.arpa.900INSOAxbru.br.ns.els-gms.att.net.
> rm-hostmaster.ems.att.com. 1179 86400 1 60 172800
>
>
>
>
> 2011/9/27 Mike Gatti 
>
>> Hello Nanog Members,
>>
>> We have been having some issue doing reverse lookups for ip's in the
>> 12.54.94.0/23 prefix. We know that this block is assigned to AT&T and
>> AT&T has assigned that block to Siemens Medical (based on whois queries). We
>> are now trying to find out who would be the authoritative DNS server that
>> would resolve PTR queries for these IP addresses. Would someone from AT&T
>> (or Siemens) or someone that has that info please contact me offline to
>> discuss?
>>
>> Thanks Everyone,
>> --
>> Michael Gatti
>> cell.949.735.5612
>> ekim.it...@gmail.com
>> (UTC-8)
>>
>>
>>
>>
>>
>>
>

Re: facebook spying on us?

[Keegan Holley]

Well what's making the connection?  It looks like unencrypted http, if your
social security number and last known addresses are streaming by you should
be able to see them.  It's a bit of a jump to say that FB (not that I'm
particularly fond of them) is spying on you from a single netstat command.
You probably clicked login with facebook for some site and it's just
autologging you in or overzealous prefetching.  Either way, I think we can
all stop making tinfoil hats now...


2011/9/29 Glen Kent 

> Hi,
>
> I see that i have multiple TCP sessions established with facebook.
> They come up even after i reboot my laptop and dont login to facebook!
>
> D:\Documents and Settings\gkent>netstat -a | more
>
> Active Connections
>
>  Proto  Local Address  Foreign AddressState
>  TCPgkent:3974www-10-02-snc5.facebook.com:http  ESTABLISHED
>  TCPgkent:3977www-11-05-prn1.facebook.com:http  ESTABLISHED
>  TCPgkent:3665
> a184-84-111-139.deploy.akamaitechnologies.com:http  ESTABLISHED
>
> [clipped]
>
> Any idea why these connections are established (with facebook and
> akamaitechnologies) and how i can kill them? Since my laptop has
> several connections open with facebook, what kind of information is
> flowing there?
>
> I also wonder about the kind of servers facebook must be having to be
> able to manage millions of TCP connections that must be terminating
> there.
>
> Glen
>
>
>

Re: SP / Enterprise design (dis)similarities

[Keegan Holley]

2011/10/10 Tom Lanyon 

> Hi all,
>
> Looking for some advice or experience in a small enterprise / hosting
> provider context.
>
> There's plenty of BCP information around for SPs in the network design
> realm, and I'm curious how much of this applies to enterprises too.
>  Commonly advised items like:
>* pull-up statics created on core devices, not network border
> devices
>* using iBGP to carry customer prefixes, not an IGP
>* announcing defaults over iBGP or IGP
>
> It depends on the size of the enterprise as well as the service provider.


> In some cases I imagine it may be simpler to have all BGP finish at the
> network border devices and not have to worry about running both IGP and iBGP
> sessions inwards to the core and/or aggregation devices.


In alot of cases iBGP is easier than an IGP if you have a large number of
routes or a large number of routers.  In others it's easier to accept only a
default from the SP and maybe a local table.  This is highly subjective as
well.


> I understand the limitations of putting our Internet prefixes in an IGP,
> but for a hosting provider style network where everything is ethernet
> connected and within data centres there's much less route flapping to deal
> with (there's no bouncing DSL lines, for example).
>

This depends as well.  There are pleanty of hosting customers that want a
full or partial table which means you'll have to run BGP.  There are all
sorts of other limitations with and IGP as well.

>
> In the case that there is both iBGP and IGP running internally, is there
> any reason to choose one or the other to originate a default route to our
> aggregation/access layers?  At some point I imagine it's going to be
> redistributed into the IGP (or re-originated in the IGP), so would think it
> would be best to just always run the default in the IGP to keep things
> consistent.
>

The more route aggregation the larger the potential for loops.  Then again
this is subjective as well.  In general BGP isn't something to be avoided at
all costs even at a small scale.  It's much easier to scale a topology that
is iBGP based than to try to move to an iBGP setup after it has grow too
large for the IGP.

>
> Finally - are there any reasons to avoid running next-hop-self on ibgp
> sessions?


I would go the other way and ask if there is any reason to advertise all the
next-hop address into the IGP?


> The upside is we get to avoid distributing all of our transit/peer upstream
> point to point links into the rest of the network.  Again, I understand this
> may be undesirable from a SP perspective, but when our 'clients' are all a
> bunch of internal servers it makes sense to keep iBGP/IGP as clean as
> possible...
>

The definition of clean is also subjective.  There are many who would run
the IGP only for loopbacks and /30's and force everything into BGP even at
small scale.  BGP makes it easier to control the routing relationships
between companies and pretty much removes the need for redistribution.
There are trade-offs though, such as load-balancing.

It really depends on your environment though and the preference of your
engineering team.  In general there isn't really a such thing as SP vs.
Enterprise.  For example I've know of a large bank with an internal MPLS
topology and about 8 public AS's used just to segregate traffic by country
of origin.  By most definitions this is an enterprise network although a
very large one.  On the other hand I've seen a service provider who's
network consisted of 4 M10i's and a firewall in someone else's colo
facility.  You have to look at which protocols and services you need to make
your network work as well as the ability to scale.  That should dictate your
design more than an imaginary line between SP and enterprise.


>
>

Re: SP / Enterprise design (dis)similarities

[Keegan Holley]

2011/10/11 Christopher Morrow 

> On Tue, Oct 11, 2011 at 1:12 AM, Keegan Holley
>  wrote:
> > The definition of clean is also subjective.  There are many who would run
> > the IGP only for loopbacks and /30's and force everything into BGP even
> at
> > small scale.  BGP makes it easier to control the routing relationships
> > between companies and pretty much removes the need for redistribution.
> > There are trade-offs though, such as load-balancing.
>
> just loadbalance toward the next-hop, no?
>

It depends on the IGP, whether the paths have exactly the same metric and
whether or not you need to run MPLS.

Re: [routing-wg] BGP Update Report

[Keegan Holley]

+1

good to get a view from multiple sources even if they are automated.  Should
be easy enough to filter for those that do not want them.

2011/10/15 William F. Maton Sotomayor 

> On Sat, 15 Oct 2011, Lynda wrote:
>
>  On 10/15/2011 4:26 AM, Geoff Huston wrote:
>>
>>> While I am at it, does anyone read this report, or is this weekly report
>>> also just part of the spam load on this list?
>>>
>>
>> I read both of them, and also the Weekly Routing Report. I will regret the
>> loss, and consider all three to be far more valuable than 90% of the traffic
>> on the list.
>>
>
> +1
>
> The reports are also useful to do a double-check on changes I've made from
> the perspective of others (even if they are automated tools).
>
> wfms
>
>
>

Re: 4.2.2.2 acting up? or is it just me?

[Keegan Holley]

I can hit it from home (comcast) and from my company's network.


2011/10/19 brian nikell 

> same
>
> On Wed, Oct 19, 2011 at 11:15 AM, Lorell Hathcock  >wrote:
>
> > All:
> >
> >
> >
> > I am experiencing trouble with reaching 4.2.2.2 right now from my
> netblock.
> > ASN 23077.
> >
> >
> >
> > Is it just me or are others experiencing the same thing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Lorell
> >
> >
> >
> >
> >
> >
>
>
> --
> -B
>
>

SaudiTelecom

[Keegan Holley]

Despite this being a north american list anyone know how I can speak with
someone from saudi telecom.  Preferably someone with the ever illusive clue?

Re: Colocation providers and ACL requests

[Keegan Holley]

Depends on the provider.  Many just do not want to manage hundreds of
customer ACL's on access routers.  Especially when it would compete with a
managed service (firewall, IDP, DDOS) of some sort.  Some still are under
the impression that ACL's are software based and their giant $100k+ edge box
would crash if they configured them for any reason.

2011/10/25 Christopher Pilkington 

> Is it common in the industry for a colocation provider, when requested to
> put an egress ACL facing us such as:
>
>  deny udp any a.b.c.d/24 eq 80
>
> …to refuse and tell us we must subscribe to their managed DDOS product?
>
> -cjp
>
>
>
>

Re: Colocation providers and ACL requests

[Keegan Holley]

2011/10/25 Brandon Galbraith 

> On Tue, Oct 25, 2011 at 1:46 PM, Keegan Holley 
> wrote:
>
>> Depends on the provider.  Many just do not want to manage hundreds of
>> customer ACL's on access routers.  Especially when it would compete with a
>> managed service (firewall, IDP, DDOS) of some sort.  Some still are under
>> the impression that ACL's are software based and their giant $100k+ edge
>> box
>> would crash if they configured them for any reason.
>>
>>
> Conversely, some don't want to be paid for bare colocation (at bare
> colocation prices) and have to then support 1000+ rules (yes, 1000+) with
> 10-20 change requests per day. YMMV/slippery slope/service scope/etc.
>

They are no worse than route filters or bandwidth increases, or IP address
requests/changes.  The provider should be able to do a temporary filter if
the customer needs it though rather than forcing them to wait weeks or
months to install a managed service/device.  I agree permanent custom ACL's
with indefinite growth/management could be considered a managed service and
should either be an additional charge or not allowed at all.

Re: Colocation providers and ACL requests

[Keegan Holley]

I'm assuming colo means hosting, and the OP misspoke.  Most colo providers
don't provide active network for colo (as in power and rack only) customers.

2011/10/25 Paul Graydon 

> On 10/25/2011 08:43 AM, Christopher Pilkington wrote:
>
>> Is it common in the industry for a colocation provider, when requested to
>> put an egress ACL facing us such as:
>>
>>   deny udp any a.b.c.d/24 eq 80
>>
>> …to refuse and tell us we must subscribe to their managed DDOS product?
>>
>> -cjp
>>
>>
>>  For colo?  No, filtering is the customers concern, unless failure to do
> so is causing a problem for the colo network.  Such services are almost
> always paid for add-ons to a colo package.  The colocation business is
> usually fairly low on the profit margin with most companies trying to get
> away with the bare minimum possible over and above the basics.
>
>
>

Re: Colocation providers and ACL requests

[Keegan Holley]

2011/10/25 Jay Ashworth 

> - Original Message -
> > From: "Keegan Holley" 
>
> > I'm assuming colo means hosting, and the OP misspoke. Most colo providers
> > don't provide active network for colo (as in power and rack only)
> customers.
>
> Most?
>
> I'm sure there are exceptions to that rule.  It's better than "YMMV".

Re: Colocation providers and ACL requests

[Keegan Holley]

2011/10/26 Jay Ashworth 

> - Original Message -
> > From: "Keegan Holley" 
>
> > > - Original Message -----
> > > > From: "Keegan Holley" 
> > >
> > > > I'm assuming colo means hosting, and the OP misspoke. Most colo
> > > > providers
> > > > don't provide active network for colo (as in power and rack only)
> > > customers.
> > >
> > > Most?
> >
> > I'm sure there are exceptions to that rule. It's better than "YMMV".
>
> Perhaps I look at a different category of colo provider, then, but I'm
> accustomed to seeing it be well up into double-digit percentage of the ones
> I've ever looked at.
>
> "Hosting", to me, means "provider's hardware", not just "local blended
> bandwidth".
>
>
> I think you may have misunderstood me. I mean local blended bandwidth to be
a colo provider offering extra services.  Hosting is provider hardware and
there should be a certain level of quality to the services and operation.  A
colo provider providing the same service as either courtesy access or a low
cost alternative to access from an ISP wouldn't be held to the same standard
for obvious reasons.  There's also "virtual hosting" which can be nothing
other than "local blended bandwidth".  But none of those webfarm types would
be on a list like this right?? ;)

Re: Route server: Route-server.ip.att.net

[Keegan Holley]

Did you do a show ip route for 12.122.83.91?  It's probably a loopback of
the nearest BGP peer it may not be the actual next hop interface IP
though.  Not sure about the blocked hops, but I can think of a few
explanations.  Overall the point of that router is to provide a view of the
route table and not the physical hops from one point to another.  Since
actual customer traffic wouldn't flow through the route server the first
few hops are probably irrelevant.

2011/11/4 Michael Sabino 

> Hi,
>
> Could you give me the relevant configs explaining why when I traceroute to
> 12.83.43.9 on route-server.ip.att.net, the first hop is "
> j6300.cbbtier3.att.net (12.0.1.202)". However, when I type "show ip route
> 12.83.43.9", the RIB shows, "* 12.122.83.91, from 12.122.83.91, 7w0d ago".
>
> I asked someone who is knowledgeable about the matter, and he seems to
> think that you can change the interface which sends back ICMP unreachables,
> but I don't know how to do this on my own simulated equipment.
>
> Also, I have noticed that when I traceroute to any ip address on the
> internet from my home connection, the last hop that's in common with all
> traceroutes is 12.83.43.9. This is a hop after several hops which seem to
> be filtered. What is the purpose of this IP?
>
> Are there any publically available documentation that would help me
> understand the process of aggregating multiple DSLAMs, etc on my at&t
> u-verse connection?
>
> I am a CCNA/CCNP student in college and this would help me understand WANs
> better.
>
>
> Thanks
>
> Michael R. Sabino
> michael.rocco.sab...@gmail.com
>
>

Re: Bandwidth Upgrade

[Keegan Holley]

Start with why you think it's necessary and what happens if mgt doesn't
listen.  Bandwidth is like electricity in a sense.  Either you have what
you need or you go belly up until some utility company can give you more
juice.  If you notice a growth pattern and are trying to get in front of it
that's obviously important.  If you are approaching the point where a
single link in redundant pairs can't handle the full load during an outage,
that's important as well.  Knowing how to use power point helps too (avoid
animation).  Maybe I don't understand the question but this is usually a
no-brainer especially if there is an existing capacity management strategy.


2011/11/17 Bielawa, Daniel Walter 

> Greetings,
>My team is in the process of putting some documentation
> together to justify a bandwidth upgrade. I am asking if you would be
> willing to reply back to me, with how you decide that it is time to upgrade
> your bandwidth. On-line or off-line reply's will be acceptable.
>
> Thank You
>
> Daniel Bielawa
> Network Engineer
> Liberty University Network Services
>
> (434)592-7987
>
> LIBERTY UNIVERSITY
> 40 Years of Training Champions for Christ: 1971-2011
>
>
>

Re: Bandwidth Upgrade

[Keegan Holley]

That depends on the network configuration though.  If you have redundant
links and one link is at 65% and the other is at 35% or more you won't be
able to get through a circuit flap or outage without dropping packets.


2011/11/17 Karl Clapp 

> Ideally, when our 95th-percentile hits 65% utilization, we begin the
> pricing and planning process and its up on peoples radar. Once the
> 95th-percentile hits 80-85% we start planning the maintenance and execute
> the upgrades. I say ideally, because in a perfect world this would happen
> 100% of the time.
>
> We try to upgrade when the 95th is at 80-85%, because the 95th-percentiles
> is based off 5-min polls, so I am sure traffic is spiking higher at peak
> times.
>
> Cheers..
>
> ~Karl
>
> On Thu, Nov 17, 2011 at 10:30 AM, Bielawa, Daniel Walter <
> dwbiel...@liberty.edu> wrote:
>
> > Greetings,
> >My team is in the process of putting some documentation
> > together to justify a bandwidth upgrade. I am asking if you would be
> > willing to reply back to me, with how you decide that it is time to
> upgrade
> > your bandwidth. On-line or off-line reply's will be acceptable.
> >
> > Thank You
> >
> > Daniel Bielawa
> > Network Engineer
> > Liberty University Network Services
> >
> > (434)592-7987
> >
> > LIBERTY UNIVERSITY
> > 40 Years of Training Champions for Christ: 1971-2011
> >
> >
>
>

Re: economic value of low AS numbers

[Keegan Holley]

Besides standing at the water cooler at 1:23PM on 12/3 telling AS123 jokes
I'm not sure a particular AS number has any relevance or any monetary value
unless there is scarcity.


2011/11/17 Kevin Loch 

> Dave Hart wrote:
>
>> AS path geeks:
>>
>> At the risk of invoking ire and eliciting comparisons to the
>> widely-reviled and growing practice of selling IPv4 addresses, I'm
>> wondering if anyone has sold legacy AS numbers for quick cash.
>>
>
> I have heard first hand stories of folks being offered 5 figures
> for four digit ASN's in the past (and they did not sell btw).   That was
> before ARIN started recycling unused short ASN's two years ago.  There
> was a three month period in 2009 where almost every ASN assigned by ARIN
> was < 1 as they burned through the backlog.
>
> - Kevin
>
>
>

Re: economic value of low AS numbers

[Keegan Holley]

2011/11/17 David Conrad 

> On Nov 17, 2011, at 8:16 AM, Keegan Holley wrote:
> > Besides standing at the water cooler at 1:23PM on 12/3 telling AS123
> jokes
> > I'm not sure a particular AS number has any relevance or any monetary
> value
> > unless there is scarcity.
>
> You are discounting (pun intended) vanity and marketing.  I am no longer
> surprised at what people will be willing to pay (sometimes astonishing
> amounts of) money for.
>
> I suppose I can't argue with that, but anyone technical enough to know
what an AS is should know better.  Also, would it really count?  What if I
opened a small ISP in some carrier hotel and paid 1000 bucks for AS 1.  I'm
not sure I'd want to sign a contract with someone dumb enough to think I
was the first company on the internet.

Re: economic value of low AS numbers

[Keegan Holley]

2011/11/17 Dave Hart 

> On Thu, Nov 17, 2011 at 18:55, Keegan Holley 
> wrote:
> > I suppose I can't argue with that, but anyone technical enough to know
> > what an AS is should know better.  Also, would it really count?  What if
> I
> > opened a small ISP in some carrier hotel and paid 1000 bucks for AS 1.
>  I'm
> > not sure I'd want to sign a contract with someone dumb enough to think I
> > was the first company on the internet.
>
> Did you intend to say the first autonomous system number assigned for
> use on ARPAnet?
>
> Pedantically yours,
>
>
I have to admit I wasn't sure.  I know ARPAnet predated BGP and EGP I
believe so I wasn't sure if there were ASN's then.

Re: Looking for a Tier 1 ISP Mentor for career advice.

[Keegan Holley]

2011/11/21 

> On Sun, 20 Nov 2011 21:40:08 EST, Tyler Haske said:
>
> > I'm looking for a mentor who can help me focus my career so eventually I
> > wind up working at one of the Tier I ISPs as a senior tech. I want to
> > handle the big pipes that hold everyone's data.
>
> OK, so I'm not a mentor from a Tier-1, and I don't directly monkey with
> routers
> as part of $DAYJOB.  But anyhow... :)
>
> With great power comes great responsibility.  Be prepared for high stress
> levels. ;)
>
> Also, keep in mind that unless you're insanely brilliant, three things
> will happen
> before you get experienced enough to be a senior tech at a Tier 1:
>
> 1) You will have grey hair (at least some).
>
> Not at all required.. Although you may grow a few belt loops and maybe
ruin a marriage or two trying to get there early.  Also, don't forget to
read, cert guides, config guides, websites, RFC's.  Grey hair and wisdom
aren't mutually inclusive.




> 3) You'll have learned that handling a big pipe at a Tier 1 isn't all
> there is
> to running a network - and in fact, quite often the Really Cool Toys are
> elsewhere.  Sure, they may have the fastest line cards, but they're going
> to
> tend to lag on feature sets just because you *don't* want to deploy
> cutting-edge code if you're a Tier-1.


Totally agree.  I touch alot of routers some of them close to what  Tier-1
would use.  I also have a few friends that work in large ISP's.  I'd say
their ultimate goal is to touch a little as possible which is usually as
unglamorous as it sounds.  Also, alot of things are scripted so much of
what you touch may not be as fun.


> As an example, AS1312 deployed IPv6 over
> a decade before some of the Tier 1's could even *spell* it (find out why
> 6bone
> existed - it's instructive history).  I'm sure that MPLS didn't make its
> first
> appearance in TIer-1 core nets either.  And the list goes on.. (Hint -
> where
> did the Tier 1's get the IPv6/MPLS/whatever experienced engineers to guide
> their deployment? :)
>

Also, how many junior and mid-level guys leave a Tier I for a network where
they can touch things and then come back as experts.  Also, the
intermediate job tends to pay for certs and training which is a plus.

Re: Odd router brokenness

[Keegan Holley]

2011/11/23 Saku Ytti 

> On (2011-11-23 09:41 -0500), Mark Radabaugh wrote:
>
> > The question is:   How does a router break in this manner?It
> > appears to unintentionally be doing something different with traffic
> > based on the source address, not the destination address.I
> > realize this can be done intentionally  - but that is not the case
> > here (unless somebody isn't telling me something).
>
> I don't think we can determine that it has anything to do with source
> address based on data shown.
> 38.104.148.5 could very well be 6500 and somehow broken adjacency to
> 74.125.226.6, perhaps hardware adjacency having MTU of 0B, causing punt
> which is rate-limited by different policer than TTL exceeded policer.
>
>
> Agree.  I've seen similar effects with a different ISP who had one side of
an ether-channel go south without the port showing down.  Stuff hashed over
the good like was fine, stuff hashed over the bad like wasn't.  Led to some
painful support calls from customers.  I agree this list is a haven of
speculation and OT comments.  In order to avoid making a bad problem worse
you should probably contact cogent.

Re: Network device command line interfaces

[Keegan Holley]

I may have a different opinion here, but I not sure I'd call any CLI easy
to work with.  Cisco's training machine is so efficient that some learn IOS
before leaving high school, so the fact that we all consider IOS easy to
work with is relative.  Just look at the "router" command.  Most of us know
that this is cisco's way of enabling protocols, but I would hardly call
this intuitive if I didn't know it already.  Then it's different for each
protocol. So "router BGP #" starts the BGP process and sets your local AS
number (very important). "router eigrp #" starts eigrp and sets a different
AS number that doesn't really count (also important). "router ospf #" just
sets a process ID in case you want to run multiple instances.  There's also
a config mode autonomous-system command but that only counts if your
running EGP which is still in the CLI but isn't supported and doesn't
start.  Then there's all the different things you can/must do with
access-lists because they were too lazy to code a different sort of
filter.  Remember CBAC?  Did I mention this is the CLI we like?  I don't
mind wrestling with a new CLI because it's all relative.  Most have read at
least one cisco book and probably one juniper book so those CLI's are
considered standard and all their sins are forgiven.  Most of us have not
gone through, training with extreme, enterasys, 3COM, netgear, foundry,
fortigate, etc. etc. etc.  So those become the PITA CLI's and suddenly
non-standard commands and bad help menus become a crime again.  I do find
text-based menus obnoxious, unless it's a linux box and the text menu is a
curses interface.  In that case it's super-cool and I'm even willing to
play games with text based menus.


2011/11/23 Jonathon Exley 

> Does anyone else despair at the CLIs produced by networking vendors?
> Real routers use a CLI that is command based, like IOS, TiMOS or Junos.
> These interfaces work well over low bandwidth connections (unlike web
> interfaces), can work with config backup systems like RANCID, have a
> (mostly) consistent structure and good show commands.
> However vendors of low cost routers/switches/muxes seem to take a stab in
> the dark and produce some really nasty stuff. I have a personal hate of
> text based menus and binary config backup files.
> Doe this p*** off anyone else? The business part of the company says "This
> device is great! It's cheap and does everything." However the poor sap who
> is given the task to make it work has to wrestle with a badly designed user
> interface and illogical syntax.
> Maybe the vendors need some sort of best practices guide for what
> manageability features their kit needs to support to make them acceptable
> to the market. Does anyone know if there is anything along these lines?
>
>
> Jonathon.
>
>
> This email and attachments: are confidential; may be protected by
> privilege and copyright; if received in error may not be used, copied, or
> kept; are not guaranteed to be virus-free; may not express the views of
> Kordia(R); do not designate an information system; and do not give rise to
> any liability for Kordia(R).
>
>

Re: Network device command line interfaces

[Keegan Holley]

That's kinda what I was talking about.  That command isn't that bad actually.  
MQC and juniper firewall filters (in set mode) are no simpler. The annoying 
part is the obscurity. 

Sent from my iPhone

On Nov 24, 2011, at 11:38 PM, Jonathon Exley  
wrote:

> Yeah, I guess Cisco IOS isn't that good an example of a consistent syntax. 
> Others do it better - Junos sets the ASN with the 'routing-options 
> autonomous-system' command, and TiMOS uses 'router autonomous-system'
> 
> My rant wasn't about having to deal with new CLIs but about the lack of CLIs 
> in those devices that seem to prefer menu based UIs (text or web), and CLIs 
> that have nasty commands. Check this out:
> 
> add flow fid-5-5 EVC-30600-Data codefault enable multi swap 99968000 
> 100032000 1024 1024 5000 ctag push 15-0 stag none
> 
> Now what does that string of numbers mean? It's the Adva 825 way of 
> specifying the CIR and EIR for a flow but I can never remember what each 
> position represents.
> 
> Compare this to TiMOS:
> 
> 
>sap-ingress 93 create
> 
>description "Test LNS"
> 
>queue 1 create
> 
>rate 2000
> 
>mbs 25 kilobytes
> 
>exit
> 
> This creates a queue with max rate 2000 kbit/s and a max burst size of 25 kB. 
> It's much easier to read than the Adva config, because each parameter is 
> labelled.
> 
> The Adva CLI isn't actually all that bad, but it's possible that had their 
> developers had some sort of usability guide when they wrote the OS then they 
> might have done things better.
> 
> I was hoping that there was already some sort of usability guide around that 
> could be shown to the manufacturers with a "please read this" note attached. 
> Is anyone aware of such a thing?
> 
> 
> Jonathon.
> 
> 
> From: Keegan Holley [mailto:keegan.hol...@sungard.com]
> Sent: Friday, 25 November 2011 4:12 p.m.
> To: Jonathon Exley
> Cc: nanog@nanog.org
> Subject: Re: Network device command line interfaces
> 
> I may have a different opinion here, but I not sure I'd call any CLI easy to 
> work with.  Cisco's training machine is so efficient that some learn IOS 
> before leaving high school, so the fact that we all consider IOS easy to work 
> with is relative.  Just look at the "router" command.  Most of us know that 
> this is cisco's way of enabling protocols, but I would hardly call this 
> intuitive if I didn't know it already.  Then it's different for each 
> protocol. So "router BGP #" starts the BGP process and sets your local AS 
> number (very important). "router eigrp #" starts eigrp and sets a different 
> AS number that doesn't really count (also important). "router ospf #" just 
> sets a process ID in case you want to run multiple instances.  There's also a 
> config mode autonomous-system command but that only counts if your running 
> EGP which is still in the CLI but isn't supported and doesn't start.  Then 
> there's all the different things you can/must do with access-lists because 
> they were too lazy to code a different sort of filter.  Remember CBAC?  Did I 
> mention this is the CLI we like?  I don't mind wrestling with a new CLI 
> because it's all relative.  Most have read at least one cisco book and 
> probably one juniper book so those CLI's are considered standard and all 
> their sins are forgiven.  Most of us have not gone through, training with 
> extreme, enterasys, 3COM, netgear, foundry, fortigate, etc. etc. etc.  So 
> those become the PITA CLI's and suddenly non-standard commands and bad help 
> menus become a crime again.  I do find text-based menus obnoxious, unless 
> it's a linux box and the text menu is a curses interface.  In that case it's 
> super-cool and I'm even willing to play games with text based menus.
> 
> 2011/11/23 Jonathon Exley 
> mailto:jonathon.ex...@kordia.co.nz>>
> Does anyone else despair at the CLIs produced by networking vendors?
> Real routers use a CLI that is command based, like IOS, TiMOS or Junos. These 
> interfaces work well over low bandwidth connections (unlike web interfaces), 
> can work with config backup systems like RANCID, have a (mostly) consistent 
> structure and good show commands.
> However vendors of low cost routers/switches/muxes seem to take a stab in the 
> dark and produce some really nasty stuff. I have a personal hate of text 
> based menus and binary config backup files.
> Doe this p*** off anyone else? The business part of the company says "This 
> device is great! It's cheap and 

Equinix

[Keegan Holley]

Assuming it's not owned by the NSA does anyone know the address of the
equnix colo in the Denver area?  I'm working on pricing access circuits
into it.  A contact from equinix would be helpful as well.  We haven't
gotten a response to our queries.

Regards,

Keegan

Re: On Working Remotely

[Keegan Holley]

Maybe I have a different personality, but I find it much easier to work
from home (provided home is empty).  I think "networking" from home, which
I do periodically during the week is different from coding from home which
I do on the weekends.  It does take some getting used to.  I find I'm much
more productive from home. (again as long as home is empty)  I spend less
time talking about sports (professional, college and little league) TV, the
opposite sex, hunting... etc. etc.  I also tend to make healthier choices
since the coffee and cigarettes aren't free and no one invites me to order
pizza for lunch when I'm at home.  To each his own though.

2011/12/4 Jay Ashworth 

> Some more thoughts on telecommuting, from the guy who built Stack Overflow.
>
> http://www.codinghorror.com/blog/2010/05/on-working-remotely.html
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land
> Rover DII
> St Petersburg FL USA  http://photo.imageinc.us +1 727 647
> 1274
>
>
>

Writable SNMP

[Keegan Holley]

For a few years now I been wondering why more networks do not use writable
SNMP.  Most automation solutions actually script a login to the various
equipment.  This comes with extra code for different vendors, different
prompts and any quirk that the developer is aware of and constant patches
as new ones come up.  Writable SNMP seems like an easier way to deal with
scripted configuration changes as well as information gathering.
Admittedly, you will have to deal with proprietary mibs and reformat the
data once it's returned.  Most people consider it insecure, but in reality
it's no worse than any other management protocol.  Yes I know netconf is
better, but in most circles I'd have problems explaining what netconf is,
why it's better and that I'm not making it up.  So I'll take what I can get.

Re: Writable SNMP

[Keegan Holley]

2011/12/6 Christopher Morrow 

> On Tue, Dec 6, 2011 at 11:16 AM, Jared Mauch 
> wrote:
> >
> > On Dec 6, 2011, at 11:07 AM, Keegan Holley wrote:
> >
> >> For a few years now I been wondering why more networks do not use
> writable
> >> SNMP.  Most automation solutions actually script a login to the various
> >> equipment.  This comes with extra code for different vendors, different
> >> prompts and any quirk that the developer is aware of and constant
> patches
> >> as new ones come up.  Writable SNMP seems like an easier way to deal
> with
> >> scripted configuration changes as well as information gathering.
> >> Admittedly, you will have to deal with proprietary mibs and reformat the
> >> data once it's returned.  Most people consider it insecure, but in
> reality
> >> it's no worse than any other management protocol.  Yes I know netconf is
> >> better, but in most circles I'd have problems explaining what netconf
> is,
> >> why it's better and that I'm not making it up.  So I'll take what I can
> get.
> >
> > Some of the problems is the bulk nature of some config changes (or
> transactional
> > nature on those that support atomic operations) have been harder to
> automate.
> >
> > Anyone that has spent any quantity of time with ASN.1 generally would
> agree.
> >
> > I recall some bay networks gear you could only program with the proper
> OID
> > as the cli was basically a SNMP-SET operation on the device.
>
> yea... same with cascade devices... icky things (both bay and cascade!)
>
> > The errors/feedback tends to be very poor over SNMP as well as you may
> need
> > to walk/revisit a large number of elements to make things happen
> properly.
>
> fun story/fact, you can send an snmp write to the broadcast address of
> a network of NT (at least, probably also post-nt versions of the OS)
> machines, and set their default-ttl to some arbitrary number. "Your
> network is too chatty... not anymore! now your internet is 5 hops
> across!"
>

Let's leave the legacy boxes out of this.  Remember that SNMP bug that
could keel over a cisco router?  I forget the details other than the guy
who wrote c code at black hat to kill routers after cisco refused to patch.

>
> > Have you had a good experience with using SNMP-Write?  I have not.
>
> long ago, in a network far away (not on the interwebs) we used snmp
> write to trigger a tftp config load. It worked nicely... I'm fairly
> certain I'd not do this on an internet connected network today though.
>

I can see the other comments about interactive commands and bulk
read/writes, but what's the harm of doing it on internet connected boxes
vs. non-internet boxes.  Just about everyone uses snmp reads in the
interwebs and as such filters it at their edges and hopefully on each
platform.  You'd secure it the same way you'd secure readable SNMP I
assume.

>
> Also, who tests snmp WRITE in their code? at scale? for daily
> operations tasks?


lol, that could be a problem.


> ... (didn't the snmp incident in 2002 teach us
> something?)
>
> Please elaborate.

> -chris
>
>

Re: Writable SNMP

[Keegan Holley]

>
> > There's no reason one can't program a device with SNMP, the main issue
> IMHO
> > has always been what I dubbed "config drift".  You have your desired
> > configuration and variances that happen over time.  If you don't force
> > a 'wr mem' or similar event after you trigger a 'copy tftp run'
> operation,
> > you may have troubles that are not apparent if there is a power failure
> > or other lossage.  The boot-time parser doesn't interpret SNMP, it parses
> > text.  This and other reasons have made people fail-safe to using the
> language
> > most easily interpreted by the device.
>
> Yup, I think the OP was maybe getting at:
>  "Why can't I snmp configure my cisco/juniper/alteon device?"
>
> I took that to mean (probably naively?) that they also would validate
> configs and update drift out of the configuration. You CAN force a 'wr
> mem' via snmp as well, of course (in cisco world).
>

It was more curiosity.  I'm looking in to scripting and starting to get
tired of having to account for ssh/telnet, credentials, differences in
platforms and code from the same vendor and my various failed attempts to
do all of the above.  Most of the automation suites I've seen work via
logins, rancid,HP NA etc etc.  Although there are better programmers that
can and have made it work it still seems cumbersome to me. I've pretty much
made the assumption that writable SNMP was a bad idea but have never
actually tried it.  I was curious what others were using, netconf or just
scripted logins. I'm also fighting a losing battle to convince people that
netconf isn't evil.  It strikes me as odd that if I wanted to talk to a
database/website full of credit card and billing info there's a long list
of API's I could use, but if I wanted to talk to the router or firewall in
front of it I can only use ssh or telnet.

Re: Writable SNMP

[Keegan Holley]

>
>
> > I can see the other comments about interactive commands and bulk
> > read/writes, but what's the harm of doing it on internet connected boxes
> vs.
> > non-internet boxes.  Just about everyone uses snmp reads in the interwebs
>
> I think the general feeling is that snmp is udp so it's spoofable and
> your perimeter acls will never be 100% (or rather, are you willing to
> risk them not being 100%?)
>

That's a given even though there's still the string to deny the spoofed
traffic someone could cause a fair amount of trouble if they knew what your
acl's look like.  That being said I don't think I've ever seen a company
that doesn't at least use SNMP for billing and basic monitoring and many
don't think to block it at their edge.  It's hard to convince someone to
change something that's been around for years though.  SNMP is flawed
though and enabling writes just makes it worse.

>
> > and as such filters it at their edges and hopefully on each platform.
> You'd
> > secure it the same way you'd secure readable SNMP I assume.
>
> read is 'painful', write can be downright deadly (to your SLAs).
>
> >>
> >> Also, who tests snmp WRITE in their code? at scale? for daily
> >> operations tasks?
> >
> >
> > lol, that could be a problem.
>
> yea, I bet the number of people that test, at scale/operations-pace,
> snmp WRITE is countable on a single hand.
>
> >>
> >> ... (didn't the snmp incident in 2002 teach us
> >> something?)
> >>
> > Please elaborate.
>
> <
> http://www.cisco.com/warp/public/707/cisco-sa-20010227-ios-snmp-ilmi.shtml
> >
>
> oh, 2001... memory lag :(
>

I don't remember hearing about this causing issues in a production network.
 According to the article you can just filter SNMP by IP which should be in
place anyway.  It's triggered by some sort of hidden string which would
make it malicious, unless I'm missing something.

In lieu of a software upgrade, a workaround can be applied to certain IOS
releases by disabling the ILMI community or "**ilmi*" view and applying an
access list to prevent unauthorized access to SNMP. Any affected system,
regardless of software release, may be protected by filtering SNMP traffic
at a network perimeter or on individual devices.

Re: Writable SNMP

[Keegan Holley]

>
>
> > assumption that writable SNMP was a bad idea but have never actually
> tried
> > it.  I was curious what others were using, netconf or just scripted
> logins.
> > I'm also fighting a losing battle to convince people that netconf isn't
> > evil.  It strikes me as odd that if I wanted to talk to a
> database/website
> > full of credit card and billing info there's a long list of API's I could
> > use, but if I wanted to talk to the router or firewall in front of it I
> can
> > only use ssh or telnet.
>
> sad, right? there are millions of restful program writers, only a few
> thousand network device programmers, and the vast majority of 'network
> management' is done by people perfectly happy with 'cisco-works' :(
>
>
according to the law of supply and demand, that would be demand? right?
;)

Re: Writable SNMP

[Keegan Holley]

2011/12/9 Joel jaeggli 

> On 12/9/11 18:22 , Keegan Holley wrote:
> >>
> >>
> >>> assumption that writable SNMP was a bad idea but have never actually
> >> tried
> >>> it.  I was curious what others were using, netconf or just scripted
> >> logins.
> >>> I'm also fighting a losing battle to convince people that netconf isn't
> >>> evil.  It strikes me as odd that if I wanted to talk to a
> >> database/website
> >>> full of credit card and billing info there's a long list of API's I
> could
> >>> use, but if I wanted to talk to the router or firewall in front of it I
> >> can
> >>> only use ssh or telnet.
>
> http://www.juniper.net/support/products/netconf/
>

+1
sadly there are still those that think netconf is black magic hacker voodo
though

Re: Writable SNMP

[Keegan Holley]

>
>
> > In lieu of a software upgrade, a workaround can be applied to certain IOS
> > releases by disabling the ILMI community or "*ilmi" view and applying an
> > access list to prevent unauthorized access to SNMP. Any affected system,
> > regardless of software release, may be protected by filtering SNMP
> traffic
> > at a network perimeter or on individual devices.
>
> right, but as I said above, the community-string restrictions don't
> help you in cases where you haven't filtered source-addresses in
> loopback/copp :( people still get to grind on your router's snmp
> process, maybe there's another way in, maybe there's a bug in the
> snmpd :(
>
> even if you filtered you could still get spoofed traffic.  What if some
employee wrote code to trace route across your network and send spoofed
packets with or without a good string.  Provided you aren't filtering snmp
at your edge, which many don't they could pretty easily melt your network
with a few boxes.  This is true of the ever present snmp poll as well.
(conspiracy theory over)

Re: Sad IPv4 story?

[Keegan Holley]

Sent from my iPhone

On Dec 10, 2011, at 2:58 AM, Randy Bush  wrote:

>> I just had a personal email from a brand new ISP in the Asia-Pacific
>> area desperately looking for enough IPv4 to be able to run their
>> business the way they would like…
> 
> and we are supposed to be surprised or feel sorry?  you're kidding,
> right?  they're lucky to be in a/p.  at least they can get a /22.
> 
> i especially like the "the way they would like" part.  the way i would
> like to run my business is to go into the office every friday and scoop
> up the cash that fell from the sky all week.
> 
> reality is such a pain in the ass.
> 
> randy
> 
+1 aren't we way past all of the predicted exhaustion dates.  There are slot of 
as's that have ignored this.
> 

Re: Sad IPv4 story?

[Keegan Holley]

2011/12/10 

> On Sat, Dec 10, 2011 at 03:15:01AM -0500, Keegan Holley wrote:
> >
> >
> > Sent from my iPhone
> >
> > On Dec 10, 2011, at 2:58 AM, Randy Bush  wrote:
> >
> > >> I just had a personal email from a brand new ISP in the Asia-Pacific
> > >> area desperately looking for enough IPv4 to be able to run their
> > >> business the way they would likeb &
> > >
> > > and we are supposed to be surprised or feel sorry?  you're kidding,
> > > right?  they're lucky to be in a/p.  at least they can get a /22.
> > >
> > > i especially like the "the way they would like" part.  the way i would
> > > like to run my business is to go into the office every friday and scoop
> > > up the cash that fell from the sky all week.
> > >
> > > reality is such a pain in the ass.
> > >
> > > randy
> > >
> > +1 aren't we way past all of the predicted exhaustion dates.  There are
> slot of as's that have ignored this.
> > >
> >
>
> predictions are ... predictions!  guesses.  swag.   nothing
> more/less.
>
>i will say this however.  after fifteen years, I am exhausted
> listening to
>
>ipv6 v. ipv4  bickering.   (and after five years of running native
> ipv6-only
>
>networks - i've re-introduced ipv4 to the mix... go figure)
>
> /bill
>
> I see your point.  The world was supposed to end dozens of times as well.
 Sorry to hear you had to reintroduce v4.  I suppose if dinosaurs were
still around we'd have to capitulate to them too.  The people who see a
T-rex and say "hey I thought they were extinct?!" would just get eaten. but
I digress. I'm not sure I'd open a new ISP at this point and expect to get
any respectable amount of IP space from the RIR right now.

Re: Your Christmas Bonus Has Arrived

[Keegan Holley]

Do the blocks have to come from a company I still work for?  If not I have
a boat load..


2011/12/13 IPv4 Brokers 

> Do you have subnets that are  not in use, or only used for specific
> purposes?  If so, please contact us.
>
> We are paying up-front (or escrow) for the use of networks that are not
> used.  The networks are used for honeypots and other research.
>
> You do not have to modify your BGP announcements, establish a GRE tunnel to
> our network and forward packets over the tunnel.
>
> The networks may be used for a month or longer, you are paid an agreed upon
> price per each month of use.
>
> Your confidentiality is absolutely guaranteed.  Only you will know that
> you're making money on your un-used or single/special-use networks.
>
> We do require a minimum of /24.
>
>

Re: Your Christmas Bonus Has Arrived

[Keegan Holley]

... Heh

>
> ipv4brok...@gmail.com
>
> -.-
>
> If domain squatting and patent trolling are both legitimate sometimes
multi-million dollar businesses are you really surprised?

Re: Range using single-mode SFPs across multi-mode fiber - was Re: NANOG Digest, Vol 47, Issue 56

[Keegan Holley]

> > inappropriate. We are attempting to use Juniper single-mode SFPs (LX
> > variety) across multi-mode fiber. Standard listed distance is always
> > for SFPs using the appropriate type of fiber. Does anyone out there
> > know how much distance we are likely to get? Thanks for your help in
> > advance.
>

Single mode just has a smaller core size for the smaller "beam" emitted by
laser vs. LED.  it works although I've never done it outside of a lab (MM
is cheaper). As for the distance it theory that should come down to the
optics and your transmit power.  Hopefully this is just a cable connecting
the router to a long line.  I've never heard of a 10K MM fiber run since SX
optics can't shoot that far.  You should be able to get through the 500m or
so that MM optics are rated for, but YMMV (errors, light levels, bounces,
etc etc)

Re: Range using single-mode SFPs across multi-mode fiber - was Re: NANOG Digest, Vol 47, Issue 56

[Keegan Holley]

2011/12/14 Justin M. Streiner 

> On Wed, 14 Dec 2011, Keegan Holley wrote:
>
>  inappropriate. We are attempting to use Juniper single-mode SFPs (LX
>>>> variety) across multi-mode fiber. Standard listed distance is always
>>>> for SFPs using the appropriate type of fiber. Does anyone out there
>>>> know how much distance we are likely to get? Thanks for your help in
>>>> advance.
>>>>
>>> Single mode just has a smaller core size for the smaller "beam" emitted
>> by
>> laser vs. LED.  it works although I've never done it outside of a lab (MM
>> is cheaper). As for the distance it theory that should come down to the
>> optics and your transmit power.  Hopefully this is just a cable connecting
>> the router to a long line.  I've never heard of a 10K MM fiber run since
>> SX
>> optics can't shoot that far.  You should be able to get through the 500m
>> or
>> so that MM optics are rated for, but YMMV (errors, light levels, bounces,
>> etc etc)
>>
>
> In a nutshell, don't do it if at all possible.  This issue gets
> significantly
> worse at 10G.  If there's any way to get SMF in place for this link, do it.
>
> +1 probably should have added that.  I guess I just assumed.


> In practice, you will likely get something less than the rated distance,
> particularly if the MM fiber in question is an older type, such as OM1. If
> you're using OM1, mode-conditioning jumpers at both ends are pretty much a
> must.
>
> The problems with shooting an LX/LH beam over MMF are threefold:
> 1. Attenuation on some flavors of MMF can be significantly higher than an
> equivalent run of SMF.
>
+1 Assumed again..


> 2. Modal dispersion on MMF will scatter and distort the LX beam, likely
> resulting in link errors because the receiver can't recover the data
> correctly.
>

Not that I'm advocating this, but it's fine over short distances.  I did
this for a few lab routers where I wasn't concerned with link quality, but
I was able to fill a 10G pipe with no errors/retransmit over about 10M.


> 3. Shooting a 9 micron beam into a 50 (or worse, 62.5) micron core, and
> getting enough of the beam to reach the 9 micron target at the other end to
> result in a recoverable signal is problematic.


Again for short distances it's doable.  I agree not to even try over 62.5
though.

Re: Range using single-mode SFPs across multi-mode fiber

[Keegan Holley]

2011/12/14 Jeff Kell 

> On 12/14/2011 3:37 PM, Keegan Holley wrote:
>
> > Single mode just has a smaller core size for the smaller "beam" emitted
> by
> > laser vs. LED.  it works although I've never done it outside of a lab (MM
> > is cheaper). As for the distance it theory that should come down to the
> > optics and your transmit power.  Hopefully this is just a cable
> connecting
> > the router to a long line.  I've never heard of a 10K MM fiber run since
> SX
> > optics can't shoot that far.  You should be able to get through the 500m
> or
> > so that MM optics are rated for, but YMMV (errors, light levels, bounces,
> > etc etc)
>
> Cisco gives specs for SFP LX over MM (they aren't that great at gig, and
> really suck at
> 10G; if you have 50u OM3/OM4 you can do much better at 10G).
>
> See SFP/fiber/distance table at
>
> http://www.cisco.com/en/US/prod/collateral/modules/ps5455/ps6577/product_data_sheet0900aecd8033f885.html
>
> They specify that line conditioning cables were used.  I would say if
you're going to bother purchasing why not purchase SM?


> We have run LX-over-MM (62.5) on short building runs as a band-aid until
> SM is
> available, and trying to do all new building MM with 50u OM3/OM4.  We do
> have some
> dependence on 62.5u MM - used by our aging Simplex alarm system - which
> does
> point-to-point looped token ring <*cough*> on the alarm side.


What distances?


>  I'm trying to get them to confirm 50u will work point-to-point, but at
> some non-alarm-points there would be a
> necessary 50-to-62.5 exchange taking place and I'm not certain how to
> accomplish that
> (50->62.5 would likely have tolerable loss, but not 62.5->50).
>
> I don't think changing core sizes in the middle would work even with SM
optics.

Re: Range using single-mode SFPs across multi-mode fiber

[Keegan Holley]

2011/12/14 oliver rothschild 

> Thanks to all who responded to my clumsy first question (both on
> matters of etiquette and technology). The group I work with (we are a
> small project acting as a last mile provider) was in the midst of
> deploying this solution when I posed the question. We put the single
> mode Juniper SFPs (LX) on to a run of approximately 1670 meters.
>

How did you end up with a MM run this long?  SX optics are only rated at
500 meters at best.  Even with mode conditioning jumpers more the 1km is a
risk.  I'm glad it held up during testing though.  Just out of curiosity
did you purchase dark from a provider?  Is it inside of a building?

Re: Range using single-mode SFPs across multi-mode fiber

[Keegan Holley]

I stand corrected, but I haven't dealt much with 100BASE-FX.  I was just
talking in terms of 1G/10G.


2011/12/14 Mark Foster 

>  On 15/12/11 16:38, Keegan Holley wrote:
>
> 2011/12/14 oliver rothschild  
>
>  Thanks to all who responded to my clumsy first question (both on
> matters of etiquette and technology). The group I work with (we are a
> small project acting as a last mile provider) was in the midst of
> deploying this solution when I posed the question. We put the single
> mode Juniper SFPs (LX) on to a run of approximately 1670 meters.
>
>
>  How did you end up with a MM run this long?  SX optics are only rated at
> 500 meters at best.  Even with mode conditioning jumpers more the 1km is a
> risk.  I'm glad it held up during testing though.  Just out of curiosity
> did you purchase dark from a provider?  Is it inside of a building?
>
>
> Um.. check that.
>
> https://en.wikipedia.org/wiki/Multi-mode_optical_fiber
>
> "Typical transmission speed and distance limits are 100 Mbit/s for
> distances up to 2 km (100BASE-FX<https://en.wikipedia.org/wiki/100BASE-FX>),
> 1 Gbit/s to 220–550 m 
> (1000BASE-SX<https://en.wikipedia.org/wiki/1000BASE-SX>),
> and 10 Gbit/s to 300 m 
> (10GBASE-SR<https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-SR>
> )."
>
> The old OM1 installations I used to work on started out as 10Mbit hubbed
> ethernet links and on the odd occasion would run out to close to 2km within
> a campus.  They were progressively upgraded with the flow of:
>
> 10FX on 3Com Linkbuilder Kit
> 100FX on 3Com Corebuilder Kit and Allied Telesyn 100FX Media converters
> 1000SX on a variety of 3Com, Nortel and Cisco kit out to ~220m
> 1000LX via Mode-Conditioning out to ~900-1000m.
>
> The OM1 only got retired when the distance was >900m or there was budget
> to put new fibre on the run, in which case we ran SMF and rigged LX drivers.
>
> Mark.
>
>
>
>
>

local_preference for transit traffic?

[Keegan Holley]

Had in interesting conversation with a transit AS on behalf of a customer
where I found out they are using communities to raise the local preference
of routes that do not originate locally by default before sending to a
other larger transit AS's.  Obviously this isn't something that was asked
of them and it took a few days to find since the customer is not a large
company and neither them nor my company has a link or business relationship
with the AS in question.  This seemed strange to me for obvious reasons,
but I was curious if anyone else was doing this and why.  You obviously
cannot use prepend to affect transit traffic again for obvious reasons.
MED is a weak metric but it at least only affects traffic that was already
going to transit your AS.  The larger transit AS was favoring a lower
bandwidth link for the customer and causing them to drop packets
mysteriously.  Just wondering if this practice seemed as strange to others
as it does to me.

Re: local_preference for transit traffic?

[Keegan Holley]

I suppose so because prepend is so easily defeated, but sometimes you don't
own a prefix shorter than the one you need to advertise.  Assuming I
understand your suggestion correctly.

2011/12/15 Holmes,David A 

> For this very reason I have advocated using longest prefix BGP routing for
> some years now, and checking periodically for the expected path, as it
> became obvious from investigating traceroutes that traffic was not being
> routed as intended using AS prepends.
>
> -Original Message-
> From: Keegan Holley [mailto:keegan.hol...@sungard.com]
> Sent: Wednesday, December 14, 2011 10:08 PM
> To: NANOG
> Subject: local_preference for transit traffic?
>
> Had in interesting conversation with a transit AS on behalf of a customer
> where I found out they are using communities to raise the local preference
> of routes that do not originate locally by default before sending to a
> other larger transit AS's.  Obviously this isn't something that was asked
> of them and it took a few days to find since the customer is not a large
> company and neither them nor my company has a link or business relationship
> with the AS in question.  This seemed strange to me for obvious reasons,
> but I was curious if anyone else was doing this and why.  You obviously
> cannot use prepend to affect transit traffic again for obvious reasons.
> MED is a weak metric but it at least only affects traffic that was already
> going to transit your AS.  The larger transit AS was favoring a lower
> bandwidth link for the customer and causing them to drop packets
> mysteriously.  Just wondering if this practice seemed as strange to others
> as it does to me.
>
> This communication, together with any attachments or embedded links, is
> for the sole use of the intended recipient(s) and may contain information
> that is confidential or legally protected. If you are not the intended
> recipient, you are hereby notified that any review, disclosure, copying,
> dissemination, distribution or use of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender immediately by return e-mail message and delete the original and
> all copies of the communication, along with any attachments or embedded
> links, from your system.
>
>

Re: local_preference for transit traffic?

[Keegan Holley]

 I always assumed that taking in more traffic was a bad thing.  I've heard
about one sided peering agreements where one side is sending more traffic
than the other needs them to transport. Am I missing something?  Would this
cause a shift in their favor allowing them to offload more customer traffic
to their peers without complaint?

2011/12/15 Jeff Wheeler 

> On Thu, Dec 15, 2011 at 1:07 AM, Keegan Holley
>  wrote:
> > Had in interesting conversation with a transit AS on behalf of a customer
> > where I found out they are using communities to raise the local
> preference
>
> That sounds like a disreputable practice.
>
> While not quite as obvious, some large transit ASes, like Level3,
> reset the origin to I (best) sometime between when they learn it and
> when they announce it to their customers and peers.  This similarly
> causes them to suck in a bit more traffic than they might otherwise.
>
> --
> Jeff S Wheeler 
> Sr Network Operator  /  Innovative Network Concepts
>
>
>

Re: local_preference for transit traffic?

[Keegan Holley]

2011/12/15 Mark Tinka 

> On Thursday, December 15, 2011 10:42:37 PM Leo Bicknell
> wrote:
>
> > However, there may be a simpler explanation.  If you bill
> > by the bit as a transit provider it's in your best
> > interest to make sure your customer gets as many bits
> > through you as possible.  Plus if you can fill their
> > pipe, they need to buy an upgrade to you.
>
> Indeed.
>
> Forgive my ignorance, but are connections between ISP's normally billed by
the bit?  I'm a transit AS but not an ISP in the traditional sense, so I
just have the normal monthly billing.

Savvis Route Server/Looking Glass

[Keegan Holley]

Does anyone know of a working Savvis route server or looking glass.  The
http://as3561lg.savvis.net/lg.html site doesn't seem to be able to query
BGP routes.  For example it says they don't have a route to 12.0/9 which
seems to be a pretty common aggregate.  The traceroute tool works normally
though.

Re: IPTV and ASM

[Keegan Holley]

Isn't source discovery and efficiency a big concern for ASM?  If individual
streams are tied to a specific source then it's possible to live without
some of the overhead involved in ASM.  Joins go straight to the source,
traffic is disseminated via direct paths instead of being replicated by the
RP, etc etc..

Disclaimer: Other than being a lab rat I haven't done much with multicast
in the wild.


2011/12/29 Jeff Tantsura 

> Mike,
>
> To my knowledge in most today's networks even if legacy equipment don't
> support IGMPv3 most likely 1st hop router does static translation and SSM
> upstream.
> The reason not to migrate to SSM is usually - ASM is already there and
> works just fine :)
> Cost to support RP infrastructure is usually the main non-technical factor
> to not to use ASM.
> Would be interested to hear from the SPs on the list.
>
> Regards,
> Jeff
>
> On Dec 28, 2011, at 2:19 PM, "Mike McBride"  wrote:
>
> > Marshall,
> >
> > On Wed, Dec 28, 2011 at 1:50 PM, Marshall Eubanks
> >  wrote:
> >> Dear Mike;
> >>
> >> On Wed, Dec 28, 2011 at 4:48 PM, Mike McBride 
> wrote:
> >>> Anyone using ASM (versus SSM) for IPTV? If so why?
> >>>
> >>
> >> From what I understand, the answer is likely to be "yes" and the
> >> reason is likely to be "deployed equipment only
> >> supports IGMP v2."
> >
> > Agreed. I'm seeking confirmation, from IPTV implementers, that non
> > igmpv3 support is the reason for using ASM with IPTV. Versus other
> > reasons such as reducing state. Or is this a non issue and everyone is
> > using SSM with IPTV?
> >
> > thanks,
> > mike
> >
> >> Regards
> >> Marshall
> >>
> >>> thanks,
> >>> mike
> >>>
> >
>
>
>

Re: Notifying customers of upstream modifications

[Keegan Holley]

Most transit networks have some sort of blanket notification that they can
send to customers.  Something like between 12AM and 6AM sometime next week
you may or may not have a moderate or severe impact, but we're not going to
give you details.  It also depends on the peering that is being added or
removed.  The larger providers are mostly static.  I can't imagine Level 3
permanently depeering from Verizon for example.  Also, if paths change but
latency and hop count are still acceptable most customers will not notice
the change.  The same goes for outages.  Also, where do you draw the line.
For example if someone severs a peering with a content network like google
some of their downstreams will care others will not.  If ISP's notified
everyone of every change it would more or less become spam so I can see an
argument for both.  In large transit networks it probably comes down to the
predicted impact of the a particular change versus visibility and
contractual liabilities.


2011/12/28 Andy Susag 

> Hi All,
>
>
>
> Just a quick question for those of you running ISPs with BGP
> downstreams.
>
>
>
> If you add or remove an upstream provider to your network, do you
> provide notification to your downstream customers? Likely, it would
> cause a shift in their traffic. If they are peering with multiple ISPs
> themselves, they may see a traffic flux.
>
>
>
> I know for a fact that our upstreams do not notify us of events so we
> tend to not send out these sort of notifications. Just wonder what
> everyone else does or if anyone happens to know "best practice"
>
>
>
> Thanks,
>
> Andy
>
>
>
>
>

Polling Bandwidth as an Aggregate

[Keegan Holley]

Has anyone had to aggregate bandwidth data from multiple interfaces
for billing.  For example I'd like to poll with an open source tool
and aggregate data from multiple interfaces connected to the same
customer or multiple customers for the purpose of billing and capacity
management.  Is there an easy way to do this with cacti/rrd or another
open source kit?

Keegan Holley ▪ Network Architect  ▪ SunGard Availability Services ▪
401 North Broad St. Philadelphia, PA 19108 ▪ (215) 446-1242 ▪

keegan.hol...@sungard.com Keeping People and Information Connected® ▪
http://www.availability.sungard.com/
Think before you print

CONFIDENTIALITY:  This e-mail (including any attachments) may contain
confidential, proprietary and privileged information, and unauthorized
disclosure or use is prohibited.  If you received this e-mail in
error, please notify the sender and delete this e-mail from your
system.

Re: Polling Bandwidth as an Aggregate

[Keegan Holley]

Thanks all for the responses.  I think I'm going to use cacti and plugins
to aggregate.  Aggregated billing is kind of something that would be nice
to have but wasn't required.  It's nice to know there are concerns with
using cacti for this.  My last question is if there is any easy/automated
way to pull interfaces into cacti and configure graphs for them either via
SNMP or reading from a mysql DB.  I suddenly remember how much I hate
importing large routers into cacti and configuring the graphs.



2012/1/20 Leo Bicknell 

> In a message written on Fri, Jan 20, 2012 at 12:16:14AM -0600, Jimmy Hess
> wrote:
> > Except Cacti/RRDTOOL is really just a great visualization tool, while you
> > can build stacks, it is not something that accurately meters data for
> > billing purposes.   The right kind of tool to use would be a netflow or
> > network tap-based billing tool,  that  actually meters/samples specific
> > datapoints at a specific interval and applies the billing business logic
> > for reporting based on sampled data points,  instead of  smoothed
> averages
> > of approximations.
>
> To suggest Netflow is more accurate than rrdtool seems rather strange
> to me.   It can be as accurate, but is not the way most people
> deploy it.
>
> RRDTool pulls the SNMP counters from an interface and records them to a
> file.  With no aggregation, and assuming your device has accurate SNMP,
> this should be 100% accurate.  While you are right that the defaults for
> RRDTOOL aggregate data (after a day, week, and month, approximately)
> those aggregates can be disabled keeping the raw data.  I know several
> ISP's that keep the raw data and use it for billing using these tools.
>
> Netflow often suffers right at the source.  If you want to bill off
> netflow data 1:1 netflow is almost required, while most ISP's do sampled
> Netflow at 1:100 or 1:1000.  Those sampling levels produce more
> inaccuracy than RRDTool's aggregation function.  What's more, once the
> data is put into the Netflow collector, they all do aggregation as well,
> just like RRDTool.  Again, you can disable much of it with careful
> configuration.
>
> But let's compare apples to apples.  Let's consider RRDTool configured
> to not aggregate with 1:1 netflow configured to not aggregate.  RRDTool
> polls a monotonically increasing counter.  Should a poll be missed no
> data is lost about the total number of bytes transferred.  Thus you can
> bill by the number of bytes transferred with 100% accuracy, even with
> missed polls.  If you bill by the bit-rate, you can interpolate a single
> missing data point which high accuracy as well.
>
> Netflow is a continuous stream of UDP across the network.  If a UDP
> packet is lost between the router and the collector there is no way to
> reconstruct that data, and it is lost forever.  Thus any network events
> means you won't have the data to bill your customer, and you're pretty
> much stuck always underbilling them with the data actually collected.
>
> > If data is not gathered using a mechanism that communicates timestamp to
> > the poller, datapoints will still be imprecise, SNMP would be an example
> > --  the cacti application may assume the SNMP response is current data,
> but
> > possibly on the actual hardware, the internal MIB on the device was
> > actually updated 10 seconds ago,  which means there will be  small spikes
> > in traffic rate graphs that do not represent actual spikes in traffic.
>
> Most of the large ISP's I know of moved away from both of the solutions
> above to propretary, custom solutions.  They SNMP poll the counters and
> store that data in a database with high resolution counters, forever,
> never aggregated.  The necessary perl/python/ruby code to do that and
> stick it in mysql or postgres is only a few pages long and easy to
> audit.
>
> --
>   Leo Bicknell - bickn...@ufp.org - CCIE 3440
>PGP keys at http://www.ufp.org/~bicknell/
>

Re: Polling Bandwidth as an Aggregate

[Keegan Holley]

 Is there a plugin for MRTG that allows you to go back to specific times?
I like MRTG better for this as well but cacti's graphs are much more
flexible.


2012/1/20 Leo Bicknell 

> In a message written on Fri, Jan 20, 2012 at 10:36:38AM -0500, Keegan
> Holley wrote:
> > using cacti for this.  My last question is if there is any easy/automated
> > way to pull interfaces into cacti and configure graphs for them either
> via
> > SNMP or reading from a mysql DB.  I suddenly remember how much I hate
> > importing large routers into cacti and configuring the graphs.
>
> I find using MRTG is easier than Cacti for _automation_ purposes.
> It's configmaker script will generate a config file for a single
> router.  I've written about 5 different versions of a small script
> that's basically a customized config maker so the graphs get named
> with customer names or the like.  The job can be fully automated
> with a few hours of coding; run it out of Cron to rebuild your interface
> list automatically and you'll never miss a customer turn up because
> someone forgot to configure a graph.
>
> --
>   Leo Bicknell - bickn...@ufp.org - CCIE 3440
>PGP keys at http://www.ufp.org/~bicknell/
>

Re: Polling Bandwidth as an Aggregate

[Keegan Holley]

2012/1/20 Chris Adams 

> Once upon a time, Leo Bicknell  said:
> > To suggest Netflow is more accurate than rrdtool seems rather strange
> > to me.   It can be as accurate, but is not the way most people
> > deploy it.
>
> Comparing Netflow to RRDTool is comparing apples to cabinets; one is a
> source of information and one is a way of storing information.
>

I assumed he meant an RRDTool kit that creates graphs with RRDTool.
Technically, mysql is the "way of storing information".  RRDTool processes
it and has the ability to make it pretty for us humons.


>
> > RRDTool pulls the SNMP counters from an interface and records them to a
> > file.
>
> No, RRDTool stores data given to it by a front end such as MRTG,
> Cricket, Cacti, etc.  That front end can fetch data from any number of
> sources, including (but not limited to) SNMP.  RRDTool then stores
> information in its database.
>

Same as above


>
> > With no aggregation, and assuming your device has accurate SNMP,
> > this should be 100% accurate.  While you are right that the defaults for
> > RRDTOOL aggregate data (after a day, week, and month, approximately)
> > those aggregates can be disabled keeping the raw data.
>
> RRDTool does not store the raw data.  Even for 5-minute intervals, it
> adjusts the data vs. the timestamp to fit the desired interval.  Since
> you don't read every counter at the exact time of your interval, RRDTool
> is always manipulating the numbers to fit.  The only numbers that are
> not changed before storing are the timestamp and value for the most
> recent update (which get overwritten at each update); everything else is
> adjusted to fit.
>
> I think every graphing tool does this.  I pretty much ignored this though
since I was asking about aggregating data from multiple objects not
aggregating data over time.

Cheers

> --
> Chris Adams 
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
>
>
>

Re: MD5 considered harmful

[Keegan Holley]

2012/1/27 Jared Mauch :
>
> On Jan 27, 2012, at 3:52 PM, Patrick W. Gilmore wrote:
>
>> Your network, your decision.  On my network, we do not do MD5.  We do more 
>> traffic than anyone and have to be in the top 10 of total eBGP peering 
>> sessions on the planet.  Guess how many times we've seen anyone even attempt 
>> this attack?  If you guessed more than zero, guess again.
>>
>> I am fully well aware saying this in a public place means someone, probably 
>> many someones, will try it now just to prove me wrong.  I still don't care.  
>> What does that tell you?
>>
>> STOP USING MD5 ON BGP.
>
> I would generally say: If you are on a p2p link or control the network, then 
> yeah, you don't need md5.  If you are at a shared medium (e.g.: IX) I do 
> recommend it there, as it will help mitigate cases where someone can hijack 
> your session by putting your IP/ASN whatnot on the router.
>
> The threat (Attack) never became real and we've now had enough time that even 
> the slowest carriers are running fixed code.
>
> - Jared
>

I kind of agree that there isn't much of a vector here, but I don't
agree that MD5 hurts if done correctly.  Is it really that hard to
find a semi-secure place to store passwords for an entire company?
There's also the question of engineering standards.  Is it an aging
practice? Probably... Is it worth spending time to update it and train
everyone not to use it?  Probably not.  I'll be happy when the world
realizes that it's ok to let gig-e auto-negotiate.  I've never really
seen MD5 cause issues.

Re: MD5 considered harmful

[Keegan Holley]

2012/1/27 Jeff Wheeler :
> On Fri, Jan 27, 2012 at 6:35 PM, Keegan Holley
>  wrote:
>> realizes that it's ok to let gig-e auto-negotiate.  I've never really
>> seen MD5 cause issues.
>
> I have run into plenty of problems caused by MD5-related bugs.
>
> 6500/7600 can still figure the MSS incorrectly when using it.  It used
> to be possible for that particular box to send over-sized frames out
> Ethernet ports with MD5 enabled, which of course were likely to be
> dropped by the neighboring router or switching equipment (perhaps even
> carrier Ethernet equipment.)  Obviously that can be a chore to
> troubleshoot.
>
> Sometimes we choose to use it.  Sometimes we don't.
>
> --

Bugs are a different argument though.  If you could call something
harmful because a single vendor codes it wrong there would be far
fewer windows users in the world. (I know it's friday, but please no
one change the subject to OS's)

Re: ARP is sourced from loopback address

[Keegan Holley]

Even though TCP dump doesn't show it the ARP packets should have a
source mac address that is reachable on the link.  I think the reply
is unicast to that mac address regardless of the IP in the request.
Otherwise the receiving station would have to do an arp request for
the source IP in the packet before it replied, in order to reply that
station would need to have the very mapping it just requested making
the whole thing useless.   I've never seen arp sourced from a
non-local interface IP unless there was some sort of tunnel or
bridging configured, but then again I don't spend my days staring at
ARP packets so I could be missing something.


2012/1/30 Joe Maimon :
>
> Hey All,
>
> Anycast related.
>
> Is this normal behavior? Whats the workaround? Why havent I run into this
> before?
>
> 192.168.76.1 is a HSRP address on a ring of routers transiting a private non
> routed vlan to the service addresses hosted on systems that have independent
> management interfaces.
>
> Best,
>
> Joe
>
>
> root@debian31:~# ifconfig lo:0
> lo:0      Link encap:Local Loopback
>          inet addr:209.54.140.64  Mask:255.255.255.255
>          UP LOOPBACK RUNNING  MTU:16436  Metric:1
>
> root@debian31:~# ip rule list
> 0:      from all lookup local
> 32764:  from 209.54.140.0/24 lookup pbr1-exit
> 32765:  from 216.222.144.16/28 lookup pbr1-exit
> 32766:  from all lookup main
> 32767:  from all lookup default
> root@debian31:~# ip route list table pbr1-exit
> default via 192.168.76.1 dev eth1
> 192.168.34.0/24 dev eth1  scope link  src 192.168.76.16
> 192.168.76.0/24 dev eth1  scope link  src 192.168.76.16
> root@debian31:~# tcpdump -i eth1
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
>
> 11:08:09.053943 ARP, Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
> 11:08:10.035126 IP noc08rt08.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 517, seq 0, length 80
> 11:08:10.051276 ARP, Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
> 11:08:11.052548 ARP, Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
> 11:08:12.035964 IP noc08rt08.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 517, seq 1, length 80
> ^C
>
> root@debian31:~# ip neigh
> fe80::230:71ff:fe3b:6808 dev eth0 lladdr 00:30:71:3b:68:08 router STALE
> 192.168.76.1 dev eth1  FAILED
> 192.168.34.254 dev eth0 lladdr 00:11:93:04:7a:1b DELAY
> 192.168.34.48 dev eth0 lladdr 00:0c:29:fd:64:8a STALE
>
> root@debian31:~# uname -a
> Linux debian31 3.2.0-1-686-pae #1 SMP Tue Jan 24 06:09:30 UTC 2012 i686
> GNU/Linux
>
> root@debian31:~# ping 192.168.76.1
> PING 192.168.76.1 (192.168.76.1) 56(84) bytes of data.
> 64 bytes from 192.168.76.1: icmp_req=1 ttl=255 time=2.95 ms
> ^C
> --- 192.168.76.1 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 2.952/2.952/2.952/0.000 ms
> root@debian31:~# ip neigh
> fe80::230:71ff:fe3b:6808 dev eth0 lladdr 00:30:71:3b:68:08 router STALE
> 192.168.76.1 dev eth1 lladdr 00:00:0c:9f:f0:01 REACHABLE
> 192.168.34.254 dev eth0 lladdr 00:11:93:04:7a:1b REACHABLE
> 192.168.34.48 dev eth0 lladdr 00:0c:29:fd:64:8a STALE
> 192.168.76.2 dev eth1 lladdr 00:b0:4a:9e:54:00 STALE
> root@debian31:~# !tcp
> tcpdump -i eth1
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
> 11:12:22.476479 IP noc08rt08-l0.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 518, seq 0, length 80
> 11:12:22.476572 IP 209.54.140.64 > noc08rt08-l0.noc08.chl.net: ICMP echo
> reply, id 518, seq 0, length 80
> 11:12:22.479495 IP noc08rt08-l0.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 518, seq 1, length 80
> 11:12:22.479533 IP 209.54.140.64 > noc08rt08-l0.noc08.chl.net: ICMP echo
> reply, id 518, seq 1, length 80
> 11:12:22.484346 IP noc08rt08-l0.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 518, seq 2, length 80
> 11:12:22.484392 IP 209.54.140.64 > noc08rt08-l0.noc08.chl.net: ICMP echo
> reply, id 518, seq 2, length 80
> 11:12:22.487670 IP noc08rt08-l0.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 518, seq 3, length 80
> 11:12:22.487705 IP 209.54.140.64 > noc08rt08-l0.noc08.chl.net: ICMP echo
> reply, id 518, seq 3, length 80
> 11:12:22.490639 IP noc08rt08-l0.noc08.chl.net > 209.54.140.64: ICMP echo
> request, id 518, seq 4, length 80
> 11:12:22.490675 IP 209.54.140.64 > noc08rt08-l0.noc08.chl.net: ICMP echo
> reply, id 518, seq 4, length 80
> ^C
>
>
>
>

Re: MD5 considered harmful

[Keegan Holley]

I suppose so but BFD certainly has alot more moving parts then adding
MDF checksums to an existing control packet.  I'm not saying everyone
should turn it on or off for that matter.  I just don't see what the
big deal is.  Most of the shops I've seen have it on because of some
long forgotten engineering standard.


2012/1/30 John Kristoff :
> On Fri, 27 Jan 2012 15:52:41 -0500
> "Patrick W. Gilmore"  wrote:
>
>> Unfortunately, Network Engineers are lazy, impatient, and frequently
>> clueless as well.
>
> While the quantity of peering sessions I've had is far less than
> yours, once upon a time when I had tried to get MD5 on dozens of peering
> sessions I learned quite a bit about those engineers and those
> networks.  I got to find out who couldn't do password management, who
> never heard of MD5 and who had been listening to Patrick.  :-) All good
> input that inform what else I might want to do to protect myself from
> those networks or who I wouldn't mind having a business relationship
> with.
>
> John
>
>

Re: ARP is sourced from loopback address

[Keegan Holley]

That's still a different part of the packet.  Below is the source
address in the ethernet header used to deliver the arp request itself.
 In side the ARP payload there is also a field for source and
destination mac.  I couldn't get tcpdump to show it even with the -n
and -vvv switches.  Wireshark will show it though.  You may be able to
use -w and -s0 to save to a cap file and then look at arp in
wireshark.  There still seem to be no responses.  You can try the
tweaks suggested by others.  I've sent traffic from a loopback before
and I've never seen this problem though.


2012/1/30 Joe Maimon :
> Thanks for the reply.
>
> Yes, it does appear to have the correct mac.
>
>
> root@debian31:~# tcpdump -e -n -i eth1
>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
> 12:54:17.882537 00:03:fd:03:38:08 > 00:0c:29:b8:2a:14, ethertype IPv4
> (0x0800), length 114: 69.90.15.224 > 216.222.144.24: ICMP echo request, id
> 161, seq 4, length 80
> 12:54:18.084320 00:0c:29:b8:2a:14 > ff:ff:ff:ff:ff:ff, ethertype ARP
> (0x0806), length 42: Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
> 12:54:19.083580 00:0c:29:b8:2a:14 > ff:ff:ff:ff:ff:ff, ethertype ARP
> (0x0806), length 42: Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
> 12:54:19.838376 00:03:fd:03:38:08 > 00:0c:29:b8:2a:14, ethertype IPv4
> (0x0800), length 407: 69.90.15.224.179 > 216.222.144.24.60714: Flags [P.],
> seq 4062306194:4062306547, ack 170308540, win 16365, length 353: BGP,
> length: 353
> 12:54:20.083649 00:0c:29:b8:2a:14 > ff:ff:ff:ff:ff:ff, ethertype ARP
> (0x0806), length 42: Request who-has 192.168.76.1 tell 209.54.140.64, length
> 28
>
> ^C
>
>
> root@debian31:~# ifconfig eth1
> eth1      Link encap:Ethernet  HWaddr 00:0c:29:b8:2a:14
>          inet addr:192.168.76.16  Bcast:192.168.76.255  Mask:255.255.255.0
>
>
>
>
> Keegan Holley wrote:
>>
>> Even though TCP dump doesn't show it the ARP packets should have a
>> source mac address that is reachable on the link.  I think the reply
>> is unicast to that mac address regardless of the IP in the request.
>> Otherwise the receiving station would have to do an arp request for
>> the source IP in the packet before it replied, in order to reply that
>> station would need to have the very mapping it just requested making
>> the whole thing useless.   I've never seen arp sourced from a
>> non-local interface IP unless there was some sort of tunnel or
>> bridging configured, but then again I don't spend my days staring at
>> ARP packets so I could be missing something.
>>
>>
>> 2012/1/30 Joe Maimon:
>>>
>>>
>>> Hey All,
>>>
>>> Anycast related.
>>>
>>> Is this normal behavior? Whats the workaround? Why havent I run into this
>>> before?
>>>
>>> 192.168.76.1 is a HSRP address on a ring of routers transiting a private
>>> non
>>> routed vlan to the service addresses hosted on systems that have
>>> independent
>>> management interfaces.
>>>
>>> Best,
>>>
>>> Joe
>>>
>>>
>>> root@debian31:~# ifconfig lo:0
>>> lo:0      Link encap:Local Loopback
>>>          inet addr:209.54.140.64  Mask:255.255.255.255
>>>          UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>>
>>> root@debian31:~# ip rule list
>>> 0:      from all lookup local
>>> 32764:  from 209.54.140.0/24 lookup pbr1-exit
>>> 32765:  from 216.222.144.16/28 lookup pbr1-exit
>>> 32766:  from all lookup main
>>> 32767:  from all lookup default
>>> root@debian31:~# ip route list table pbr1-exit
>>> default via 192.168.76.1 dev eth1
>>> 192.168.34.0/24 dev eth1  scope link  src 192.168.76.16
>>> 192.168.76.0/24 dev eth1  scope link  src 192.168.76.16
>>> root@debian31:~# tcpdump -i eth1
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
>>>
>>> 11:08:09.053943 ARP, Request who-has 192.168.76.1 tell 209.54.140.64,
>>> length
>>> 28
>>> 11:08:10.035126 IP noc08rt08.noc08.chl.net>  209.54.140.64: ICMP echo
>>> request, id 517, seq 0, length 80
>>> 11:08:10.051276 ARP, Request who-has 192.168.76.1 tell 209.54.140.64,
>>> length
>>> 28
>>> 11:08:11.052548 ARP, Request who-has 192.168.76.1 tell 209.54.140.64,
>>> length
>>> 28
>>&g

Re: Hijacked Network Ranges

[Keegan Holley]

You can break your blocks into /24's or smaller and readvertise them to
your upstreams.  You can also modify local preference using community tags
with most upstreams.  If you have tier 1 peerings you may be able to get
them to filter the bad routes if you can prove they were assigned to you by
ARIN.  There's no real way to get 100% of your traffic back until you get
the other company to stop advertising your routes though.  You may also get
traction from the AS's directly connected to the problem AS.  I'm not sure
how quickly you can get the other AS's to act on your behalf.  The short
blocks and local pref should get some of your traffic back though.


2012/1/31 Kelvin Williams 

> Greetings all.
>
> We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet
> Exchange) immediately filter out network blocks that are being advertised
> by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.
>
> The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and
> 68.66.112.0/20 are registered in various IRRs all as having an origin AS
> 11325 (ours), and are directly allocated to us.
>
> The malicious hijacking is being announced as /24s therefore making route
> selection pick them.
>
> Our customers and services have been impaired.  Does anyone have any
> contacts for anyone at Cavecreek that would actually take a look at ARINs
> WHOIS, and IRRs so the networks can be restored and our services back in
> operation?
>
> Additionally, does anyone have any suggestion for mitigating in the
> interim?  Since we can't announce as /25s and IRRs are apparently a pipe
> dream.
>
> --
> Kelvin Williams
> Sr. Service Delivery Engineer
> Broadband & Carrier Services
> Altus Communications Group, Inc.
>
>
> "If you only have a hammer, you tend to see every problem as a nail." --
> Abraham Maslow
>
>

Re: Hijacked Network Ranges

[Keegan Holley]

2012/1/31 Justin M. Streiner 

> On Tue, 31 Jan 2012, Grant Ridder wrote:
>
>  What is keeping you from advertising a more specific route (i.e /25's)?
>>
>
> Many providers filter out anything longer (smaller) than /24.
>

Some will accept it but not propagate it upstream.  This may be useful in
redirecting all the traffic from a large AS if you are directly connected.


>
> jms
>
>
>  On Tue, Jan 31, 2012 at 12:00 PM, Kelvin Williams > >wrote:
>>
>>  Greetings all.
>>>
>>> We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek
>>> Internet
>>> Exchange) immediately filter out network blocks that are being advertised
>>> by ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.
>>>
>>> The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and
>>> 68.66.112.0/20 are registered in various IRRs all as having an origin AS
>>> 11325 (ours), and are directly allocated to us.
>>>
>>> The malicious hijacking is being announced as /24s therefore making route
>>> selection pick them.
>>>
>>> Our customers and services have been impaired.  Does anyone have any
>>> contacts for anyone at Cavecreek that would actually take a look at ARINs
>>> WHOIS, and IRRs so the networks can be restored and our services back in
>>> operation?
>>>
>>> Additionally, does anyone have any suggestion for mitigating in the
>>> interim?  Since we can't announce as /25s and IRRs are apparently a pipe
>>> dream.
>>>
>>> --
>>> Kelvin Williams
>>> Sr. Service Delivery Engineer
>>> Broadband & Carrier Services
>>> Altus Communications Group, Inc.
>>>
>>>
>>> "If you only have a hammer, you tend to see every problem as a nail." --
>>> Abraham Maslow
>>>
>>>
>>
>
>

Re: Hijacked Network Ranges - paging Cogent and GBLX/L3

[Keegan Holley]

To be honest I haven't had much success it convincing a tier 1 to
modify someone else's routes on my behalf for whatever reason.  I also
have had limited success in getting them to do anything quickly.  I'd
first look to modify your advertisements as much as possible to
mitigate the issue and then work with the other guys upstreams second.


2012/1/31 Schiller, Heather A :
>
> Or roll it up hill:
>
> 33611 looks like they get transit from 19181, who's only upstream appears to 
> be 12189.
> 12189 gets connectivity from 174 and 3549.
>
> 174 = Cogent
> 3549 = GBLX/L3
>
>  --Heather
>
> -Original Message-
> From: Kelvin Williams [mailto:kwilli...@altuscgi.com]
> Sent: Tuesday, January 31, 2012 1:01 PM
> To: nanog@nanog.org
> Subject: Hijacked Network Ranges
>
> Greetings all.
>
> We've been in a 12+ hour ordeal requesting that AS19181 (Cavecreek Internet
> Exchange) immediately filter out network blocks that are being advertised by 
> ASAS33611 (SBJ Media, LLC) who provided to them a forged LOA.
>
> The routes for networks: 208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 
> are registered in various IRRs all as having an origin AS
> 11325 (ours), and are directly allocated to us.
>
> The malicious hijacking is being announced as /24s therefore making route 
> selection pick them.
>
> Our customers and services have been impaired.  Does anyone have any contacts 
> for anyone at Cavecreek that would actually take a look at ARINs WHOIS, and 
> IRRs so the networks can be restored and our services back in operation?
>
> Additionally, does anyone have any suggestion for mitigating in the interim?  
> Since we can't announce as /25s and IRRs are apparently a pipe dream.
>
> --
> Kelvin Williams
> Sr. Service Delivery Engineer
> Broadband & Carrier Services
> Altus Communications Group, Inc.
>
>
> "If you only have a hammer, you tend to see every problem as a nail." --
> Abraham Maslow
>
>

Re: [#135346] Unauthorized BGP Announcements (follow up to Hijacked Networks)

[Keegan Holley]

That may not be a bad idea.  Have you gotten your company's lawyers
involved? They may be able to get some sort of court action started and get
things moving. They may also be able to compel the ISP's to act.


2012/1/31 Kelvin Williams 

> I hope none of you ever get hijacked by a spammer housed at Phoenix NAP.
>  :)
>
> We're still not out of the woods, announcing /24s and working with upper
> tier carriers to filter out our lists.  However, I just got this response
> from Phoenix NAP and found it funny.  The "thief" is a former customer,
> whom we terminated their agreement with.  They then forged an LOA,
> submitted it to CWIE.net and Phoenix NAP and resumed using space above and
> beyond their terminated agreement.  So now any request for assistance to
> stop our networks from being announced is now responded to with an
> instruction to contact the thief's lawyer.
>
> kw
>
> -- Forwarded message --
> From: Kelvin Williams 
> Date: Tue, Jan 31, 2012 at 7:43 PM
> Subject: Re: [#135346] Unauthorized BGP Announcements
> To: n...@phoenixnap.com
>
>
> We'll be forwarding this to our peers in the industry--rather funny that
> Phoenix NAP would rather send us to the attorney of the people stealing our
> space than bothering to perform an ARIN WHOIS search, or querying any of
> the IRRs.
>
> Interesting...  Very interesting...  So, who all do you have
> there--spammers and child pornographers?  Is this level of protection what
> you give to them all?
>
>
>
> On Tue, Jan 31, 2012 at 7:30 PM, Brandon S 
> wrote:
>
> > Hello,
> >
> > Thank you for your email. Please direct any further questions regarding
> > this issue to the following contact.
> >
> > Bennet Kelley
> > 100 Wilshire Blvd.
> > Suite 950
> > Santa Monica, CA 90401
> > bkel...@internetlawcenter.net
> >
> > Telephone
> > 310-452-0401
> >
> > Facsimile
> > 702-924-8740
> >
> > --
> > Brandon S.
> > NOC Services Technician
> >
> > ** We want to hear from you!**
> > We care about the quality of our service. If you’ve received
> > anything less than a prompt response or exceptional service or would like
> > to share any
> > feedback regarding your experience, please let us know by sending an
> email
> > to management:
> > supportfeedb...@phoenixnap.com
> >
> > --
> Kelvin Williams
> Sr. Service Delivery Engineer
> Broadband & Carrier Services
> Altus Communications Group, Inc.
>
>
> "If you only have a hammer, you tend to see every problem as a nail." --
> Abraham Maslow
>
>

Re: UDP port 80 DDoS attack

[Keegan Holley]

There aren't very many ways to combat DDOS.  That's why it's so popular.
Some ISP's partner with a company that offers a tunnel based scrubbing
service where they DPI all your traffic before they send it to you.  If you
only have a few upstreams it may be helpful to you.  I spoke to them last
year but we have too many links and too many blocks to use it.  I think the
name of the company was prolexic.  They're also a L3 VAR if you have L3
links.  There isn't alot of BGP (AFAIK) magic that doesn't involve cutting
someone off to save the rest of your customers.

2012/2/5 Ray Gasnick III 

> We just saw a huge flux of traffic occur this morning that spiked one of
> our upstream ISPs gear and killed the layer 2 link on another becuase of a
> DDoS attack on UDP port 80.
>
>
>
> Wireshark shows this appears to be from a compromised game server (call of
> duty) with source IPs in a variety of different prefixes.
>
>
>
> Only solution thus far was to dump the victim IP address in our block into
> the BGP Black hole community with one of our 2 providers and completely
> stop advertising to the other.
>
>
>
> Anybody see this recently and have any tips on mitigation,  reply on or
> off list.
>
>
>
> Thank You,
>
> Ray Gasnick III
> CISSP, Technology Specialist: Network Security & Infrastructure
> Miles Technologies
> www.milestechnologies.com
>
> Phone: (856) 439-0999 x127
> Direct: (856) 793-3821
> How am I doing?  Email my manager at itmana...@milestechnologies.com
> 
>
> Computer Networking – IT Support – Business Software – Website Design –
> Online Marketing & PR
>
>
>

Re: UDP port 80 DDoS attack

[Keegan Holley]

An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping,
and RTBH?  The first four will not work against a DDOS attack and the last
one just kills the patient so he does not infect other patients.  As I said
earlier beyond traffic scrubbing offsite there isn't much defense against
DDOS.

2012/2/5 Dobbins, Roland 

>
> On Feb 6, 2012, at 7:21 AM, Keegan Holley wrote:
>
> > There aren't very many ways to combat DDOS.
>
> Start with the various infrastructure/host/service BCPs, and S/RTBH, as
> outlined in this preso:
>
> <https://files.me.com/roland.dobbins/dweagy>
>
> ---
> Roland Dobbins  // <http://www.arbornetworks.com>
>
>The basis of optimism is sheer terror.
>
>  -- Oscar Wilde
>
>
>
>

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/5 Dobbins, Roland 

>
> On Feb 6, 2012, at 8:10 AM, Keegan Holley wrote:
>
> > An entire power point just to recommend ACL's, uRPF, CPP, DHCP snooping,
> and RTBH?
>
> Actually, no, that isn't the focus of the preso.
>
> > The first four will not work against a DDOS attack
>
> This is incorrect - suggest you read the preso.
>

The ACL's are configured on the routers belonging to the victim AS which
will not save their access pipe if it's overrun unless I'm missing
something.  uRPF may help with spoofed traffic, but sometimes causes
problems with multi-homing and is often more harmful than helpful depending
on the network design.

>
> > and the last one just kills the patient so he does not infect other
> patients.
>
> S/RTBH - as opposed to D/RTBH - doesn't kill the patient.  Again, suggest
> you read the preso.
>

Source RTBH often falls victim to rapidly changing or spoofed source IP"s.
It also isn't as widely supported as it should be. I never said DDOS was
hopeless, there just aren't a wealth of defenses against it.

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/5 Dobbins, Roland 

>
> On Feb 6, 2012, at 8:37 AM, Keegan Holley wrote:
>
> > Source RTBH often falls victim to rapidly changing or spoofed source
> IP"s.
>
> S/RTBH can be rapidly shifted in order to deal with changing purported
> source IPs, and it isn't limited to /32s.  It's widely supported on Cisco
> and Juniper gear (flowspec is a better choice on Juniper gear).
>
> I was referring to support from ISP's not from hardware vendors.

If folks don't want to read the presos or search through the archives,
> that's fine, of course.  The fact is that there are quite a few things that
> operators can and should do in order to mitigate DDoS attacks; and making
> the perfect the enemy of the merely good only helps the attackers, doesn't
> it?
>
> Yes but assuming everything discussed at a conference is instantly adopted
by the entire industry gives one false hope no?

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/5 Steve Bertrand 

> On 2012.02.05 20:37, Keegan Holley wrote:
>
>> 2012/2/5 Dobbins, Roland
>>
>
>  S/RTBH - as opposed to D/RTBH - doesn't kill the patient.  Again, suggest
>>> you read the preso.
>>>
>>>
>> Source RTBH often falls victim to rapidly changing or spoofed source IP"s.
>> It also isn't as widely supported as it should be. I never said DDOS was
>> hopeless, there just aren't a wealth of defenses against it.
>>
>
> This is so very easily automated. Even if you don't actually want to
> trigger the routes automatically, finding the sources you want to blackhole
> is as simple as a monitor port, tcpdump and some basic Perl.
>

This is still vulnerable to spoofing which could cause you to filter
legitimate traffic and make the problem worse.  Not saying that S/RTBH is a
bad idea.  RTBH is effective and a great idea just not very elegant.


>
> ...and as far as this not having been deployed in many ISPs (per your next
> message)... their mitigation strategies should be asked up front, and if
> they don't have any (or don't know what you speak of), find a new ISP.
>

You sometimes have to weigh the pro's and cons.  You can't always pick the
guys with the coolest knobs.

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/6 Jeff Wheeler 

> On Mon, Feb 6, 2012 at 8:43 PM, Sven Olaf Kamphuis 
> wrote:
> > there is a fix for it, it's called "putting a fuckton of ram in -most-
> > routers on the internet" and keeping statistics for each destination
> > ip:destination port:outgoing interface so that none of them individually
> can
> > (entirely/procentually compared to other traffic) flood the outgoing
> > interface on that router... end result, if enough routers are structured
> > like that, is that ddos attacks will be come completely useless.
>
> There are two obvious problems with your approach.
>
> First, adding the policers you suggest, at the scale needed, is a
> little harder than you imagine.  It's not a simple matter of the cost
> of RAM but also power/heat density per port.
>

Since when are policers implemented in ram?  You're talking FPGA if you
want to be able to make forwarding/filtering decisions assuming it's
possible which it isn't you're 1 million dollar boxes suddenly become
hundred million dollar boxes.  Then there's v6 info..

>
> Second, if you re-engineer every router on the Internet to prevent an
> interface from being congested by malicious flow(s) destined for one
> particular destination IP:port, then DDoS attacks will simply target
> multiple ports or multiple destination IP addresses that are likely to
> traverse a link they are able to congest.
>


Not to mention that the routers themselves become an attack vector.  This
cache will have a finite limit because there's no such thing as an infinite
amount of cache among other flaws.  When that limit is reached it will do
something no one want's it to do and that will become the new DDOS attack.

>
> If you want to dramatically increase the cost of routers in order to
> solve the problem of DDoS with one deft (and expensive) move, you have
> to imagine that the people behind DDoS attacks aren't complete idiots,
> and will actually spend some time thinking about how to defeat your
> system.
>
> Not to mention cost?  You're not going to get a tier I ISP to upgrade to
this new super router (assuming it's possible to build such a things)
without an act of congress or at least the FCC.  They won't even spend
enough on fiber to bring broadband into rural areas.

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/8 George Bonser 

>
>
> > -Original Message-
> > From: bas
> > Sent: Tuesday, February 07, 2012 11:56 PM
> > To: Dobbins, Roland; nanog
> > Subject: Re: UDP port 80 DDoS attack
> >
> > Say eyeball provider X has implemented automated S/RTBH, and I have a
> > grudge against them.
> > I would simply DoS a couple of the subscribers *with spoofed source IP*
> > addresses from google, youtube, netflow and hulu.
> > The automated S/RTBH drops all packets coming from those IP addresses.
> > Presto; many angry consumers call the ISP's helpdesk.
>
> Comes back to providers allowing "spoofed" traffic into their networks
> from customers.  That seems to me to be the low-hanging fruit here.
>
>
>
How do you stop it?  Granted, traffic from 10/8 or 127.0.0.1 coming in via
an upstream is obvious, but that's about it.  There's nothing in a packet
that will tell you where it came from compared to the source IP field in
the IP header.  uRPF is a problem for anyone who's sufficiently multihomed
since it causes asymmetric routing.

Re: UDP port 80 DDoS attack

[Keegan Holley]

It works in theory, but to get every ISP and hosting provider to ACL their
edges and maintain those ACL's for every customer no matter how large might
be a bit difficult.  Also, what about non-BGP customers or customers that
just accept a default route? Or even customers that just want return
traffic to come in a different link for some reason.  ISP's would suddenly
become giant traffic registries.

2012/2/8 George Bonser 

>
>
> >From: Keegan Holley
>
> >How do you stop it?
>
> A provider knows what destination IP traffic they route TO a customer,
> don't they?  That should be the only source IPs they accept FROM a customer.
>
>
> If you don't route it TO the customer, you shouldn't accept it FROM the
> customer unless you have made special arrangements with them and verified
> they are entitled to source the traffic from the desired IPs.
>
>
>
>

Re: UDP port 80 DDoS attack

[Keegan Holley]

Providers don't even check the registries for bgp advertisements. See the 
thread on hijacked routes for proof.   Not to mention how do you handle a small 
transit AS?  Do you trust that they have the correct filters as well?  Do you 
start reading their AS paths and try to filter based on the registry for folks 
down stream?  Then there's the RLDRAM issue.  Most edge boxes will just run out 
if ACL's.  Lastly there's no contractual obligation to play traffic cop for the 
entire Internet so providers would be dropping traffic that they can 
legitimately bill for.

Sent from my iPhone

On Feb 8, 2012, at 4:56 AM, George Bonser  wrote:

>> No, we have registries to act as registries, the ISPs should be
>> checking them, and double checking.  It isn't something that is going
>> to change every day or every week. Once you get it set up, it is going
>> to be stable for a while.  Sure, it means a little more work in setting
>> up a customer, but it also means that if all your neighbors do the same
>> thing, you field many fewer calls dealing with stupid DoS crap.
>> 
> 
> I'll put it another way. Any provider that does not police their customer 
> traffic has no business whining about DoS problems.
> 
> 

Re: UDP port 80 DDoS attack

[Keegan Holley]

On Feb 8, 2012, at 4:51 AM, George Bonser  wrote:

> 
> 
>> From: Keegan Holley 
>> Subject: Re: UDP port 80 DDoS attack
> 
>> It works in theory, but to get every ISP and hosting provider to ACL their 
>> edges and maintain those ACL's for every customer no matter how large might 
>> be a bit difficult.  
> 
> You don't have to ACL in most cases. RPF works for most.  There will be a 
> few, relatively darned few, that you will need to ACL, but RPF takes care of 
> a large number of them.
> 

RPF on the whole Internet would pretty much lead to an instant outage.  What 
happens when you have two upstreams and one has an incoming route to you but 
your out going route for which ever of their customers talking to you doesn't 
agree?  Instant outage.  Multiply that by the entire table and then add churn.  
I'd give it a week before everyone turned it off,  if you could even get them 
to enable it to begin with.
 
> 
>> Also, what about non-BGP customers or customers that just accept a default 
>> route?  
> 
> I don't follow.  The ISP still knows what traffic gets routed TO them.  You 
> only accept FROM them what you route TO them, even if you hand them a default 
> route.
> 
> 
>> Or even customers that just want return traffic to come in a different link 
>> for some reason.
> 
> Still don't follow.  I am not talking about having a firewall that is 
> stateful.  I am talking packet by packet.  If you don't route it to them, you 
> don't accept it from them unless you have made arrangements otherwise, which 
> will be a small percentage of your customers. Sure, some might be multihomed 
> but it is easy enough to verify that they have the networks in question 
> SWIPed to them or a call to the other provider can clear that up in a few 
> minutes.  It isn't THAT hard.
> 
> 
>> ISP's would suddenly become giant traffic registries.
> 
> 
> No, we have registries to act as registries, the ISPs should be checking 
> them, and double checking.  It isn't something that is going to change every 
> day or every week. Once you get it set up, it is going to be stable for a 
> while.  Sure, it means a little more work in setting up a customer, but it 
> also means that if all your neighbors do the same thing, you field many fewer 
> calls dealing with stupid DoS crap.
> 
> 

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/8 Dobbins, Roland 

> On Feb 8, 2012, at 8:07 PM, bas wrote:
>
> > As far as I see it S/RTBH is in no way a solution against smart
> attackers, of course it does help against all the kiddie attacks out
> > there.
>
> Once again, I've used S/RTBH myself and helped others use it many, many
> times, including to defend against attacks with shifting purported source
> IPs.  flowspec, IDMS and other tools are very useful as well, but S/RTBH is
> supported on a lot of hardware, if operators choose to configure it.
>
> It is not a panacea.  It is one tool in the toolbox.
>
> Folks can either choose to make use of it or choose not to do so; it is
> operationally proven, it does work, and it's certainly better than nothing.
>  YMMV.
>
>
I agree.  I think RTBH is a broadsword not a scalpel.  It's a tool in the
tool box and there is a danger of dropping legitimate traffic with both
S/RTBH and D/RTBH.  BGP isn't a security protocol.  It's not even that
great of a routing protocol.

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/8 George Bonser 

> > 77% of all networks seem to think so.
> > http://spoofer.csail.mit.edu/summary.php
>
> And it would be the remaining 23% that really need to understand how
> difficult they are making life for the rest of the Internet.
>

23% of 4.29 billion addresses is still more than enough to ruin anyone's
day.

Re: UDP port 80 DDoS attack

[Keegan Holley]

2012/2/8 Steve Bertrand 

> On 2012.02.08 14:23, Drew Weaver wrote:
>
>> Stop paying transit providers for delivering spoofed packets to the edge
>> of your network and they will very quickly develop methods of proving that
>> the traffic isn't spoofed, or block it altogether. =)
>>
>
> I firmly believe in this recourse, amongst others...
>

How do you tell the spoofed packets from the non-spoofed ones?  Especially
if you have more than one provider.

>
> If you know that your provider allows spoofed traffic, let the community
> know about it.
>

According to a company wide NDA I'm only allowed to disclose that to the
best of my knowledge my upstreams permits packets sent from users or other
NSP's who may or may not permit or generate packets.  The source IP
addresses are checked to be valid 32 bit numbers before being sent to my
routers. My upstreams to the best of their knowledge have never sent me a
single spoofed packet and will refrain from doing so unless they receive
written consent from me, in triplicate. ;)

>
> In all aspects of life, a problem must be 'fixed' at the source. All of
> the small-medium size ops have to connect to the big-boys somewhere, and
> what I've seen in this industry is that the big-boys are generally
> compliant.
>

As long as compliant means completely indifferent to your concerns and
unwilling to change or compromise in any meaningful while sucking money
away faster than the government.  They are all very very compliant and a
pleasure to do business with.

Re: Common operational misconceptions

[Keegan Holley]

Alot of people are unclear on how hard it is for someone to sniff internet
traffic if the aren't physically located at or near one of the endpoints
IE: connected to the  same access point or same switch.

2012/2/15 John Kristoff 

> Hi friends,
>
> As some of you may know, I occasionally teach networking to college
> students and I frequently encounter misconceptions about some aspect
> of networking that can take a fair amount of effort to correct.
>
> For instance, a topic that has come up on this list before is how the
> inappropriate use of classful terminology is rampant among students,
> books and often other teachers.  Furthermore, the terminology isn't even
> always used correctly in the original context of classful addressing.
>
> I have a handful of common misconceptions that I'd put on a top 10 list,
> but I'd like to solicit from this community what it considers to be the
> most annoying and common operational misconceptions future operators
> often come at you with.
>
> I'd prefer replies off-list and can summarize back to the list if
> there is interest.
>
> John
>
>
>

Re: time sink 42

[Keegan Holley]

If you're building a datacenter probably not.  Other than giving the remote
hands some identifier and making them label the servers themselves.  If
you're at a conference you could get away with using masking tape and a
sharpie.  If you think it was time consuming applying the labels wait until
you need to remove one.


2012/2/16 Randy Bush 

> ok, this is horribly pragmatic, but it's real.  yesterday i was in the
> westin playing rack and stack for five hours.  an horrifyingly large
> amount of my time was spent trying to peel apart labels made on my
> portable brother label tape maker, yes peeling the backing from a little
> label so remote hands could easily confirm a server they were going to
> attack.
>
> is there a trick?  is there a (not expensive) different labeling machine
> or technique i should use?
>
> randy
>
>
>

Re: Programmers with network engineering skills

[Keegan Holley]

+1 on both.  Senior network guys learn programming/scripting as a way to
automate configuration and deal with large amounts of data.  It's an
enhancement for us and most network people are willing to expand their
programming skills given the time.  On the other hand there are way too
many jobs where programmers can just be programmers for many of them to be
interested in expanding their networking skills even if they have prior
experience.  If they become interested in the "hardware" world they usually
go toward systems administrator and OS's.  Some of them are big enough
geeks to want to learn both or all three, but those are few and far
between.  It's very likely that such programmers frequent this list so
hopefully I won't get flamed for posting this.  EIther way it's just
semantics, but it is generally easier to find a network guy that wants to
learn how to program or get better at it than to find a programmer who is
dying to learn about networking.  Not sure if I agree with the opinion
about generalists.  There are alot of people who view technology as both a
job and a hobby and become experts in what pays their bills and then slowly
learn something about everything via osmosis.  There are alot of people
that never saw a book or trade rag they didn't like.


2012/2/27 Owen DeLong 

> I think you're more likely to find a network engineer with (possibly
> limited)
> programming skills.
>
> That's certainly where I would categorize myself.
>
> Owen
>
> On Feb 27, 2012, at 12:02 PM, Brandt, Ralph wrote:
>
> > Generalists are hard to come by these days. They are people who learn
> > less and less about more and more till they know nothing about
> > everything. People today are specializing in the left and right halves
> > of the bytes  They learn more and more about less and less till they
> > know everything about nothing.  And BTW, they are worthless unless you
> > have five of them working on a problem because none of them know enough
> > to fix it.  Worse, you can replace the word five with fifty and it may
> > be still true.
> >
> > I know of three of these, all gainfully employed at this time and could
> > each find at least a couple jobs if they wanted.  I am one, my son is
> > two and a guy we worked with is the third.
> >
> > At one time (40 years ago) the mantra in IS was train for expertise, now
> > it is hire for it.  Somewhere there has to be a happy medium.  I suggest
> > this, find a good coder, not a mediocre who writes shit code but a good
> > one who can think and learn and when you talk about branching out with
> > his skill set he or she lights up.  His first thing on site is take the
> > A+ networking course.
> >
> > No, I do not sell the courses.  But I have seen this kind of approach
> > work when nothing else was.
> >
> >
> >
> >
> > Ralph Brandt
> > Communications Engineer
> > HP Enterprise Services
> > Telephone +1 717.506.0802
> > FAX +1 717.506.4358
> > Email ralph.bra...@pateam.com
> > 5095 Ritter Rd
> > Mechanicsburg PA 17055
> >
> > -Original Message-
> > From: A. Pishdadi [mailto:apishd...@gmail.com]
> > Sent: Sunday, February 26, 2012 8:27 PM
> > To: NANOG
> > Subject: Programmers with network engineering skills
> >
> > Hello All,
> >
> > i have been looking for quite some time now a descent coder (c,php) who
> > has
> > a descent amount of system admin / netadmin experience. Doesn't
> > necessarily
> > need to be an expert at network engineering but being acclimated in
> > understanding the basic fundamentals of networking. Understanding basic
> > routing concepts, how to diagnose using tcpdump / pcap, understanding
> > subnetting and how bgp works (not necessarily setting up bgp). I've
> > posted
> > job listings on the likes of dice and monster and have not found any
> > good
> > canidates, most of them ASP / Java guys.
> >
> > If anyone can point me to a site they might recommend for job postings
> > or
> > know of any consulting firms that might provide these services that
> > would
> > be greatly appreciated.
>
>
>
>

Re: Programmers with network engineering skills

[Keegan Holley]

2012/3/2 Randy Bush 

> > In my experience the path of least resistance is to get a junior
> > network engineer and mentor he/she into improving his/hers programming
> > skills than go the other way around.
>
> and then the organization pays forever to maintain the crap code while
> the kiddie learned to program.  right.  brilliant.
>
> +1 Although, I've seen the opposite where a brilliant developer writes
wonderful code, leaves and you are left with a similarly difficult
situation since there are no more programmers in the department and no
brilliant developers willing to do programming that requires in depth
knowledge of networking.


> Always code as if the guy who ends up maintaining your code will be a
> violent psychopath who knows where you live. -- Martin Golding
>
> randy
>
>
>

Re: Programmers with network engineering skills

[Keegan Holley]

2012/3/2 Randy Bush 

> >>> In my experience the path of least resistance is to get a junior
> >>> network engineer and mentor he/she into improving his/hers programming
> >>> skills than go the other way around.
> >>
> >> and then the organization pays forever to maintain the crap code while
> >> the kiddie learned to program.  right.  brilliant.
> >
> > +1 Although, I've seen the opposite where a brilliant developer writes
> > wonderful code, leaves and you are left with a similarly difficult
> > situation since there are no more programmers in the department and no
> > brilliant developers willing to do programming that requires in depth
> > knowledge of networking.
>
> that was not a brilliant developer.  that was a clever developer.
>
>Debugging is twice as hard as writing the code in the first place.
>Therefore, if you write the code as cleverly as possible, you are,
>by definition, not smart enough to debug it.  -- Brian W. Kernighan
>
>
It's not so much that the code was too clever to troubleshoot, just that we
were fresh out of developers.


> and, if the department was not willing to invest in long-term software
> capability, then they were foolish to enter the game in the first place.
>

If I said this was the first time I saw a large corporation do something
foolish I'd be lying...  I was consulting on a project that created a need
to modify the existing code.  I probably could have tackled it but I have a
day job and didn't want to become the "house developer".  Watching people
try to explain to upper management why their band of merry router jockeys
should have a developer was interesting.  Sometimes it comes down to
convincing the business side to invest time and money into creating the
development position for code that hasn't been touched in years..  If you
just look at the technical bits, the need is usually obvious.


>
> go find an open-source solution or buy commercial.  and if none fit your
> needs, and you are not willing to invest in softdev, then you have a
> problem in your business model.
>

Agreed... but I was consulting.  My business model was satisfied when I
walked through the door.  I'm not saying there shouldn't be developers on a
team of network engineers, it's was just interesting to see what happens
when the one-eye'd man leaves.

Re: Programmers with network engineering skills

[Keegan Holley]

2012/3/5 Owen DeLong 

> Given my experience to date with the assumptions made by programers about
> networking in the following:
>
>Apps (iOS apps, Droid apps, etc.)
>Consumer Electronics
>Microcontrollers
>Home Routers
>
> I have to say that the strategy being used to date, whichever one it is,
> is not working. I will also note that the erroneous assumptions, incorrect
> behaviors, and other problems I have encountered with these items are
> indicative of coders that almost learned networking more than of networkers
> that almost learned software development.
>
>
I think it comes down to economics mostly.  Most development jobs either do
not require knowledge of networking or do not enforce the requirement.
There are plenty of jobs where developers do not need to know networking so
when it's a sticking point it just becomes harder to find someone that
fits.  This doesn't give the average developer much incentive to learn
networking, even if it leads to buggy or incorrectly written code.  On the
other hand a senior net-eng that can code is worth is weight in gold,
especially if he can spit out palatable webUI's for everything.

Re: Whitelist of update servers

[Keegan Holley]

2012/3/12 Maverick 

> Is there a whitelist that applications have to talk to in order to
> update themselves?
>
> sometimes

Re: Whitelist of update servers

[Keegan Holley]

2012/3/12 Maverick 

> Like list of sites that operating systems or applications installed on
> your machines go to update themselves. One way could be to go on each
> vendors site and look at their update servers like
> microsoft.update.com but it would be good if there is a list of such
> servers for all OS and applications so that it could be used as a
> whitelist.
>
>
I stick with my original answer... sometimes.  I'm not sure if this is
different now, but I remember MS update being spoofed with bogus DNS
entries because the process is died to that dns name.  I think this is the
most popular method combined with some sort of encryption and/or signing to
verify the updates themselves.  I'm sure there are applications that use a
white list though.  There are alot of shops that update via some kind of
CDN, so the whitelist method is a bit combersome at scale and is not immune
to spoofing or other attacks.  The most secure thing is probably to protect
the updates themselves.

Re: Programmers with network engineering skills

[Keegan Holley]

2012/3/12 Tei 

> On 12 March 2012 09:59, Carlos Martinez-Cagnazzo 
> wrote:
> > Hey!
> >
> > On 3/8/12 8:24 PM, Lamar Owen wrote:
> >> On Monday, March 05, 2012 09:36:41 PM Jimmy Hess wrote:
> >> ...
> >>>(16)  The default gateway's IP address is always 192.168.0.1
> >>>(17) The user portion of E-mail addresses never contain special
> >>> characters like  "-" "+"  "$"   "~"  "."  ",", "[",  "]"
> > I've just had my ' xx AT cagnazzo.name' email address rejected by a web
> > form saying that 'it is not a valid email address'. So I guess point
> > (17) can be extended to say that 'no email address shall end in anything
> > different that .com, .net or the local ccTLD'
> >
> > :=)
> >
> > Carlos
>
>
> Yea, I don't even know how programmers can get that wrong.  The regex
> is not even hard or anything.
>
>
>
> (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
>
>
I bet it's even harder without the use of a search engine.

Re: Programmers with network engineering skills

[Keegan Holley]

On Mar 12, 2012, at 5:32 PM, Owen DeLong  wrote:

> 
> On Mar 12, 2012, at 2:12 PM, Keegan Holley wrote:
> 
>> 2012/3/12 Tei 
>> 
>>> On 12 March 2012 09:59, Carlos Martinez-Cagnazzo 
>>> wrote:
>>>> Hey!
>>>> 
>>>> On 3/8/12 8:24 PM, Lamar Owen wrote:
>>>>> On Monday, March 05, 2012 09:36:41 PM Jimmy Hess wrote:
>>>>> ...
>>>>>>  (16)  The default gateway's IP address is always 192.168.0.1
>>>>>>  (17) The user portion of E-mail addresses never contain special
>>>>>> characters like  "-" "+"  "$"   "~"  "."  ",", "[",  "]"
>>>> I've just had my ' xx AT cagnazzo.name' email address rejected by a web
>>>> form saying that 'it is not a valid email address'. So I guess point
>>>> (17) can be extended to say that 'no email address shall end in anything
>>>> different that .com, .net or the local ccTLD'
>>>> 
>>>> :=)
>>>> 
>>>> Carlos
>>> 
>>> 
>>> Yea, I don't even know how programmers can get that wrong.  The regex
>>> is not even hard or anything.
>>> 
>>> 
>>> 
>>> (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
>>> 
>>> 
>> I bet it's even harder without the use of a search engine.
> 
> Whenever I've built code to check someone's email address on a form, I always 
> just looked for the following:
> 
> 1.matches ^[^@]+@[A-Za-z0-0\-\.]+[A-Za-z]$
> 2.The component to the right of the @ sign returns at least one A, , 
> or MX record.
> 
> If it passed those two checks, I figured that was about as good as I could do 
> without resorting to one of the following:
>1.An incomprehensible and unmaintainable regex as the one above
>2.Actually attempting delivery to said address
> 
> Owen
> 

I've done some scripting with the similar goals and to be completely honest 
I've skews at least consulted google.  It's much easier to read and test a 
regular expression like the one above than to write one from scratch.  
Sometimes it takes an incomprehensible regex to be thorough. Sometimes close 
enough really is close enough though. It depends on the problem you are trying 
to solve.   
> 

Re: Verizon FiOS - is BGP an option?

[Keegan Holley]

In defense of the tier 1's it's not as easy as it looks to run BGP with the
lower end business customers.  On the technical side the edge boxes and
links to them would be as overloaded with routes and peers and all of the
other PE boxes in an ISP network.  Not to mention the changes in routing
policies and addressing schemes and the general operation of the service.
This obviously isn't the case for every ISP, but I can understand why it's
not popular.

Re: Looking for some diversity in Alabama that does not involve ATT Fiber

[Keegan Holley]

I feel a topic shift coming...


2012/3/21 Jay Ashworth 

> - Original Message -
> > From: "Eric Wieling" 
>
> > I don't know about AT&T, but Verizon physically removes the copper
> > connections when they install fiber into a building. Oddly, this is
> > legal. Verizon is required to open up their copper to CLECs, but not
> > fiber.
>
> The Verizon *regulated ILEC operating company* is required to provide equal
> access.  FiOS comes from an unregulated subsidiary.
>
> Whether there might be some illegal collusion in the unreg subsid
> generating
> a pull order for a copper service from the regulated LEC is one thing...
>
> but why would it otherwise be illegal for the LEC to pull the copper?
>
> It *is* their copper...
>
> That's an interesting perception, and I'm curious where you came by it.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land
> Rover DII
> St Petersburg FL USA  http://photo.imageinc.us +1 727 647
> 1274
>
>
>

Re: last mile, regulatory incentives, etc (was: att fiber, et al)

[Keegan Holley]

2012/3/22 Jared Mauch 

>
> On Mar 22, 2012, at 11:05 AM, chris wrote:
>
> > I'm all for VZ being able to reclaim it as long as they open their fiber
> > which I don't see happening unless its by force via government. At the
> end
> > of the day there needs to be the ability to allow competitors in so of
> > course they shouldnt be allowed to rip out the regulated part and replace
> > it with a unregulated one.
>


Maybe I'm missing something, but how exactly does one share fiber?  Isn't
it usually a closed loop between DWDM or Sonet nodes?  It doesn't seem fair
to force the incumbents to start handing out lambdas and timeslots to their
competitors on the business side.  I guess passive optical can be shared
depending on the details of the network, but that would still be much
different than sharing copper pairs.