Re: Ingress filtering on transits, peers, and IX ports

[Bryan Holloway]

I too would like to know more about their methodology and actual 
tangibles ideally in the form of PCAPs.



On 10/14/20 4:56 PM, Brian Knight via NANOG wrote:

Hi Eric,

I shot a message over the folk who did the testing for more info about 
their test.  If I'm able to find anything useful in our logs from their 
detail, I'll post it to the list.


The message referenced DNS cache poisoning and DDOS amplification, so it 
seemed fairly general and more focused on whether ASes accepted spoofed 
traffic.  They also referenced the new NXNSAttack, which I did not know 
about previously.


Thanks,

-Brian

Re: Vint Cerf & Interplanetary Internet

[Bryan Holloway]

Don't mess with Belters ...

On 10/22/20 9:20 AM, C. A. Fillekes wrote:


the subgroup for networks on aspherical planetoids would be EGGNOG -- we 
only meet during the holiays


On Wed, Oct 21, 2020 at 11:59 PM Mark Andrews > wrote:


It wouldn’t be NANOG.  Perhaps LUNOG or MOONOG.

 > On 22 Oct 2020, at 14:07, scott weeks mailto:sur...@mauigateway.com>> wrote:
 >
 >
 > *From:* NANOG mailto:gmail@nanog.org>> on behalf of Rod Beck
mailto:rod.b...@unitedcablecompany.com>>
 >>

https://www.quantamagazine.org/vint-cerfs-plan-for-building-an-internet-in-space-20201021/
 > 
 >
 > On 10/21/20 2:27 PM, Suresh Ramasubramanian wrote:
 >
 > Right. This means we are going to catch a spaceship for a future
nanog / have
 > interplanetary governance federation debates with space aliens
from Andromeda,
 > and we will finally run out of v6 and ipv9 will rule the roost
while there’s a
 > substantial aftermarket + hijack scene going on for the last
remaining v6 blocks.
 > 
 >
 >
 > More like IP to Nokia's new cell network on the moon:
 >
 >

https://www.theguardian.com/science/2020/oct/20/talking-on-the-moon-nasa-and-nokia-to-install-4g-on-lunar-surface
 > (Everyone on the moon will want to have access to LOL cats!)
 >
 > Or... using DTN (https://datatracker.ietf.org/wg/dtn/about) to
reach Mars and other
 > planets by being relayed through communications relay satellites
similar to the
 > Mars Telecommunication Orbiter (canceled),  Mars Odyssey or Mars
 > Reconnaissance Orbiter spacecraft.
 >
 > Or... IP to robots visiting other non-planet objects in the solar
system like
 > comets/asteroids:
 > https://spacenews.com/osiris-rex-touches-down-on-asteroid
 > https://www.bbc.com/news/science-environment-47293317
 >
 > Or... 
 >
 > The IPI idea has been around for a long time now:
 > https://en.wikipedia.org/wiki/Interplanetary_Internet
 >
 > The main question is will NANOG On The Road meet on the moon?  I
missed
 > the only Hawaii one, so maybe I could make the moon one!
 >
 > scott

-- 
Mark Andrews, ISC

1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: ma...@isc.org

Mellanox / Cumulus

[Bryan Holloway]

Curious to hear if the community has had any real-world experience using 
Mellanox/Cumulus (nVidia) for L2/L3 things outside of the datacenter.


Like other vendors, notably Arista, they seem to be trying to move out 
of the datacenter and target SPs and the layer 3 market. Personally, I 
think Arista has worked out most of the kinks over the last few years, 
and we've been happy with their L3 solutions (e.g., the 7280s)


While Mellanox's chipset is intriguing, I get a sense of feature-itis 
from their marketing. (BGP, OSPF, NAT, we do it all etc.) No IS-IS 
support, I'm told ...


Anybody using these in production in an SP environment? And if so, any 
opinions, good or bad?


Feel free to reach out off-list if you prefer. Thank you,

- bryan

Re: Are the days of the showpiece NOC office display gone forever?

[Bryan Holloway]

"I'd piss on a spark plug if I thought it'd do any good ..."

(Ed.: I'd take that "NOC" any day.)


On 12/17/20 4:33 PM, Joe Provo wrote:

On Wed, Dec 16, 2020 at 12:49:52PM -0800, Eric Kuhnke wrote:
[snip]

Are the days of such an environment gone forever?


We can only hope so.

Re: Parler

[Bryan Holloway]

There's a pretty big difference between imparting knowledge and inciting 
violence.


#redherring

Disclaimer: I own this book.


On 1/12/21 6:40 PM, Andy Ringsmuth wrote:

And yet, Amazon will still happily sell you this item:

https://www.amazon.com/Anarchist-Cookbook-William-Powell/dp/1607966123/

In fact, it is listed as:  #1 Best Seller in Anarchism




Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas, 1863


On Jan 12, 2021, at 10:36 AM, Paul Timmins  wrote:

"You have to let your customer's services contain death threats against the owner of 
your company or we'll blacklist you" is the wildest take of 2021 yet.

Blocking Amazon because of who they allow to remain a customer is something I 
wholeheartedly encourage my competitors to do.

On 1/12/21 9:29 AM, Kevin McCormick wrote:

Imagine if Tier 1 ISPs had a censorship free clause that required companies 
like Twitter, Facebook, and Amazon to provide services free of censorship or 
have IP blocks blackholed. They would lose hundreds of millions of dollars per 
day. I bet they would reverse their tone in a hurry.

https://www.seattletimes.com/seattle-news/idaho-internet-provider-to-block-facebook-twitter-over-their-trump-bans/
  
Thank you,


Kevin McCormick
  
From: NANOG  On Behalf Of mark seery

Sent: Sunday, January 10, 2021 8:06 PM
To: K. Scott Helms 
Cc: NANOG Operators' Group 
Subject: Re: Parler
  
I assume multiple networks/ ISPs that have acceptable use policies that call out criminality and incitement to violence, for example:
  
https://www.xfinity.com/support/articles/comcast-acceptable-use-policy
  
Have these AUPs been invoked previously for these reasons, or would that be new territory?


Sent from Mobile Device


On Jan 10, 2021, at 2:52 PM, K. Scott Helms  wrote:


Right, it's not a list for content hosting.
  
Scott Helms
  
On Sun, Jan 10, 2021, 5:42 PM  wrote:

No, this is a list for Network Operators.

Sent from my iPhone


On Jan 10, 2021, at 5:32 PM, K. Scott Helms  wrote:


This is a list for pushing bits.  The fact that many/most of us have other 
businesses doesn't make this an appropriate forum for SIP issues (to use my own 
work as an example).
  
On Sun, Jan 10, 2021, 4:52 PM  wrote:

This is a list for Network Operators, AWS certainly operates networks.

Sent from my iPhone


On Jan 10, 2021, at 4:27 PM, K. Scott Helms  wrote:


No,
  
It really does not.  Section 230 only applies to publishers, and not to network providers.  If this were a cloud hosting provider list then you'd be correct, but as a network provider's list it does not belong here.



Scott Helms

  
  
On Sun, Jan 10, 2021 at 3:21 PM Lady Benjamin PD Cannon  wrote:

As network operations and compute/cloud/hosting operations continue to 
coalesce, I very much disagree with you.  Section 230 is absolutely relevant, 
this discussion is timely and relevant, and it directly affects me as both a 
telecom and cloud compute/services provider.
  
  
—L.B.
  
Lady Benjamin PD Cannon, ASCE

6x7 Networks & 6x7 Telecom, LLC
CEO
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ






On Jan 10, 2021, at 12:13 PM, K. Scott Helms  wrote:
  
It's not, and frankly it's disappointing to see people pushing an agenda here.



Scott Helms


On Sun, Jan 10, 2021 at 9:37 AM  wrote:


NANOG is a group of Operators, discussion does not have to be about networking. 
I have already explained how this represents a significant issue for Network 
Operators.

On Jan 10, 2021, at 9:09 AM, Mike Bolitho  wrote:


It has nothing to do with networking. Their decision was necessarily political. 
If you can specifically bring up an issue, beyond speculative, on how their new 
chosen CDN is somehow now causing congestion or routing issues on the public 
internet, then great. But as of now, that isn't even a thing. It's just best to 
leave it alone because it will devolve into chaos.

- Mike Bolitho

On Sun, Jan 10, 2021, 6:54 AM  wrote:


Why? This is extremely relevant to network operators and is not political at 
all.

On Jan 10, 2021, at 8:51 AM, Mike Bolitho  wrote:


Can we please not go down this rabbit hole on here? List admins?

- Mike Bolitho

On Sun, Jan 10, 2021, 1:26 AM William Herrin  wrote:


Anybody looking for a new customer opportunity? It seems Parler is in
search of a new service provider. Vendors need only provide all the
proprietary AWS APIs that Parler depends upon to function.

https://www.washingtonpost.com/technology/2021/01/09/amazon-parler-suspension/

Regards,
Bill HErrin

Hosting recommendations ... ?

[Bryan Holloway]

Hey gang ...

Looking for a reputable (i.e., no hosting of spammers or other 
ne'er-do-wells) hosting provider with possibly a global footprint. If 
not, US is #1 desire; EU #2.


Requirements, more or less:

* Desire to host 2-3 hypervisors, probably running something akin to 
Proxmox ...


* ~5-10TB storage with the possibility of expansion ...

* 1G hand-off / 100 Mbps or less commit ... i.e., low BW, but burstable.

* Bringing my own IP space and need to be able to peer BGP with vendor.

* Cross-DC redundancy or mirroring or somesuch desirable.

* Backups are of interest, although I can do my own if need be.

Any recommendations that are non-Amazonian? Feel free to reach out 
off-list if you prefer.


#1 requirement ... the reputability part ...

Thank you all in advance!

- bryan

Re: Hosting recommendations ... ?

[Bryan Holloway]

Fair questions -- answers in-line ...


On 1/19/21 5:19 PM, Josh Luthman wrote:
I'm kind of confused when your concern is the reputability and yet 
you're providing your own IP space.


I care about the hosting environment's upstreams' opinions of the 
downstream.


That is, I don't want to be in a situation where some jamoke (or 
jamokes) does something which affects me from a collateral damage 
standpoint.


It sounds more to me like you want to put 2-3 boxes in a data center.  
For that pretty much any decent sized data center in any state would 
work for the US.


I would like to stop personally dealing with bare-metal. That's what I'm 
doing now.

Re: Hosting recommendations ... ?

[Bryan Holloway]

Perhaps I'm missing something, but in your #1 example "Cloud", what 
prevents me from running a Proxmox ISO (which is more or less Debian) 
vs. a "standard" Debian install on the provider's virtual server?


If I can, I've succeeded. That is the sort of hosting provider I'm 
looking for, if they exist.


#2 would be suitable, but it seems to be that if leased bare-metal dies, 
it will be some time for ETR. Less desirable, but I'm open to ideas.


#3 I do now. Trying to move away from that.


On 1/19/21 5:44 PM, William Herrin wrote:

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway  wrote:

I would like to stop personally dealing with bare-metal. That's what I'm
doing now.


Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally
adjustable to meet your needs. You manage the operating systems and
storage within the virtual environment. You DO NOT manage the host
operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on
the equipment including any hypervisors needed to run virtual servers.
You DO NOT deal with hardware break/fix, that problem belongs to the
service provider.

Colocation = You lease space in a data center. You provide physical
equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable
for contradicting the others. So I ask again for clarification: which
of these do you seek?

Regards,
Bill Herrin

Re: Hosting recommendations ... ?

[Bryan Holloway]

You make an excellent point, Martijn ... (and I suspect this is what 
Bill was pointing out ...)


Virtualization on top of virtualization is inherently not the best idea.

I guess I'm looking for flexibility in the sense of being able to spin 
up additional VMs at my leisure. In which case #2 could be suitable in 
the right environment.


... in which case SLAs would be tantamount to success ...

So ... that said, any recommendations?  :)


On 1/19/21 6:32 PM, Martijn Schmidt wrote:
For #1, are you trying to do "Cloud-ception" e.g. running your own 
proxmox virtualization on top of an already virtual machine, so that 
you're basically two layers deep?


For #2, of course you need to be able to survive a hardware failure 
(using RAID1 or some flavour of DRBD for example) but having to think 
about such things is the "trade-off" of having access to the bare-metal 
layer.. it does have advantages, for example if you want to install your 
own virtualization layer without any involvement from the hosting 
provider. You'd usually have agreements with the hosting provider about 
how/when hardware replacements would be done.


Best regards,
Martijn
----
*From:* NANOG  on behalf 
of Bryan Holloway 

*Sent:* 19 January 2021 18:18
*To:* William Herrin 
*Cc:* NANOG list 
*Subject:* Re: Hosting recommendations ... ?
Perhaps I'm missing something, but in your #1 example "Cloud", what
prevents me from running a Proxmox ISO (which is more or less Debian)
vs. a "standard" Debian install on the provider's virtual server?

If I can, I've succeeded. That is the sort of hosting provider I'm
looking for, if they exist.

#2 would be suitable, but it seems to be that if leased bare-metal dies,
it will be some time for ETR. Less desirable, but I'm open to ideas.

#3 I do now. Trying to move away from that.


On 1/19/21 5:44 PM, William Herrin wrote:

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway  wrote:

I would like to stop personally dealing with bare-metal. That's what I'm
doing now.


Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally
adjustable to meet your needs. You manage the operating systems and
storage within the virtual environment. You DO NOT manage the host
operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on
the equipment including any hypervisors needed to run virtual servers.
You DO NOT deal with hardware break/fix, that problem belongs to the
service provider.

Colocation = You lease space in a data center. You provide physical
equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable
for contradicting the others. So I ask again for clarification: which
of these do you seek?

Regards,
Bill Herrin

Re: Hosting recommendations ... ?

[Bryan Holloway]

On 1/19/21 6:33 PM, Brandon Martin wrote:


On 1/19/21 11:44 AM, William Herrin wrote:



Cloud = you get virtual servers with virtual storage, generally
adjustable to meet your needs. You manage the operating systems and
storage within the virtual environment. You DO NOT manage the host
operating systems or hypervisors.




It's worth pointing out that nested virtualization is a thing these 
days, and some providers might even support it!  That means you could 
buy one large instance and sub-divide it yourself into multiple VMs if 
you want to.


In practice, unless you need that flexibility to dynamically spin the 
VMs up and down with various specs AND don't want to or cannot use a 
provider's API for that, I'm not sure why you'd want to if you didn't 
have to for some crazy reason.


I'm very curious about your assertion:

Is nested virtualization really a thing?

I mean, I'm not exactly trying to render Pixar's latest movie ... just 
trying to push some bits around (light web-sites, some e-mail ...)


It just seems inherently prone to issues.

Could you back this up with any white-papers or documentation on the 
subject? I'm genuinely interested ...


- bryan

Re: Hosting recommendations ... ?

[Bryan Holloway]

Thank you, everyone, for the advice, input, and suggestions, both on- 
and off-list.


Got a few sales pitches too, which was to be expected. :) All good.

Much appreciated, again.

Cheers,
- bryan



On 1/19/21 4:44 PM, Bryan Holloway wrote:

Hey gang ...

Looking for a reputable (i.e., no hosting of spammers or other 
ne'er-do-wells) hosting provider with possibly a global footprint. If 
not, US is #1 desire; EU #2.


Requirements, more or less:

* Desire to host 2-3 hypervisors, probably running something akin to 
Proxmox ...


* ~5-10TB storage with the possibility of expansion ...

* 1G hand-off / 100 Mbps or less commit ... i.e., low BW, but burstable.

* Bringing my own IP space and need to be able to peer BGP with vendor.

* Cross-DC redundancy or mirroring or somesuch desirable.

* Backups are of interest, although I can do my own if need be.

Any recommendations that are non-Amazonian? Feel free to reach out 
off-list if you prefer.


#1 requirement ... the reputability part ...

Thank you all in advance!

     - bryan

Anyone from Cloudflare peering lurking?

[Bryan Holloway]

Trying to get a few pubIX sessions up ... ping me off-list, s.v.p.?

E-mails to the usual contacts aren't working.

Thanks!

Re: AW: OVH datacenter SBG2 in Strasbourg on fire 🔥

[Bryan Holloway]

Vitrol™ -- Ask for it by name.

At $dayjob-1 I ran one of Enron's old (abandoned) datacenters which was 
built in the early aughts or earlier.


Even that had full pre-action systems, which we once triggered when one 
of my colleagues accidentally hooked up a battery backwards.


*poof*

Fire Department was there in under five minutes.

So yeah -- seems a little weird.


On 3/10/21 7:16 PM, Randy Bush wrote:

the conjecturbation is only surpassed by the vitrol

Anyone from Intuit lurking?

[Bryan Holloway]

Got some customers on certain prefixes which are unable to reach various 
*.intuit.com sites. We've checked all routing/peering and it looks like 
there may be a block in place.


If someone from Intuit is lurking, could you reach out to me off-list?

Thank you!

Office Depot contact?

[Bryan Holloway]

Howdy folks,

If anyone from Office Depot NetOps is lurking, could you please reach 
out to me off-list?


Looks like our whole AS is getting blocked somewhere ...

Thank you!
- bryan

Re: Any2 LAX

[Bryan Holloway]

This is what I got from those guys ...

--

CoreSite Incident Notification


Description:  During a planned maintenance event to integrate new 
hardware into our MPLS core an extreme dip in Any2 traffic was observed. 
After about 4 hours running in a degraded state, an emergency case was 
opened with the hardware vendor. After working with the hardware vendor 
to rule out any possible hardware or software bugs, the network 
engineering team located the source of the traffic loss. It was an 
errant configuration applied by the custom automation written to build 
LSP's in our MPLS network. A formal IR will be provided for this event.





On 6/11/21 8:03 PM, jim deleskie wrote:
Also saw a major traffic drop. There is a Root Cause to be issued early 
in the week I'm told.



-jim

On Fri, Jun 11, 2021 at 2:42 PM Siyuan Miao > wrote:


Yea, it was down but both RS are online and feeding us unreachable
nexthops during the outage .

On Sat, Jun 12, 2021 at 1:27 AM Seth Mattinen mailto:se...@rollernet.us>> wrote:

On 6/11/21 10:16 AM, Jon Lewis wrote:
 > On Fri, 11 Jun 2021, Seth Mattinen wrote:
 >
 >> Did Any2 LAX barf last night between about 1am and 8am
Pacific time?
 >
 > More like 00:00-7:45 (Pacific time).
 >
 > Anyone know what broke, and why the IX was dead for nearly 8
hours?
 > This is our second recent issue with "an Any2 IX", having
dealt with an
 > IX partition event at Any2 Denver just a few weeks ago.
 >


What I saw was a lot of unreachable nexthops (I'm in LA2) on routes
advertised through the route servers. Most of my direct BGP
sessions
were down, but a handful were still working including the route
servers.

For example, I was getting routes for AS29791 from the route
servers,
but nexthop 206.72.211.106 was dead to me. Not to pick on
Internap other
than a mutual customer called me directly at 1am and wanted to
know why
things were down.

I killed the route server sessions and went back to sleep.

Feels like LA1 and LA2 got split, but however the route servers
interconnect still worked, which was problematic.

Re: Any2 LAX

[Bryan Holloway]

On 6/11/21 8:25 PM, Seth Mattinen wrote:

On 6/11/21 11:18 AM, Bryan Holloway wrote:

This is what I got from those guys ...

--

CoreSite Incident Notification


Description:  During a planned maintenance event to integrate new 
hardware into our MPLS core an extreme dip in Any2 traffic was 
observed. After about 4 hours running in a degraded state, an 
emergency case was opened with the hardware vendor. After working with 
the hardware vendor to rule out any possible hardware or software 
bugs, the network engineering team located the source of the traffic 
loss. It was an errant configuration applied by the custom automation 
written to build LSP's in our MPLS network. A formal IR will be 
provided for this event.






Was that an automated email? Last time I got any email from Coresite was 
April 22.



Automated.

Re: The great Netflix vpn debacle!

[Bryan Holloway]

Is there some new DB that major CDNs are using?

We've been getting several reports of prefixes of ours being blocked, 
claiming to be VPNs, even though we've been using those subnets without 
incident for years.


HBO, Netflix, and Hulu appear to be common denominators. I have to 
wonder if they're all siphoning misinformation off of some new DB 
somewhere ...



On 8/14/21 1:45 AM, Mike Hammett wrote:

https://thebrotherswisp.com/index.php/geo-and-vpn/



-
Mike Hammett
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 


*From: *"John Alcock" 
*To: *nanog@nanog.org
*Sent: *Friday, August 13, 2021 2:11:16 PM
*Subject: *The great Netflix vpn debacle!

Well,

It happened. I have multiple subscribers calling in. They can not access 
Netflix.


Any contacts on list for Netflix that I can use to get my up blocks 
whitelisted?


John

Re: akamai yesterday - what in the world was that

[Bryan Holloway]

This echoed events a month or so ago, and I'm curious as to what is 
making these releases more, uh, network-impacting.


Game releases are hardly a new thing, but these last two events seem to 
be almost an order of magnitude higher than what we're used to (at least 
on our predominantly eyeball network.)


Any thoughts from the community? We're taking steps to accommodate, but 
from a capacity-planning perspective, this seems non-linear to me.


Or is it just in my head?


On 1/23/20 9:20 AM, Jared Mauch wrote:




On Jan 23, 2020, at 10:16 AM, Kaiser, Erich  wrote:

Yeah we saw that as well. Must be a game release or something.


Yes, that’s my understanding as well.

- Jared

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Bryan Holloway]

I didn't think one could get a single 'B' channel over ISDN ... but I 
could be mistaken.


In my early ISP days, ISDN was 2 x 64k (full-rate) 'B' channels and a 
16k 'D' channel for signaling.



On 1/26/20 5:58 AM, Joly MacFie wrote:

IIRC that 64k was in fact 56k with 8k for overhead.

I had one, and it would kick in a second channel if you pushed it, for a 
whopping 112k. Metered, came out to about $500/mo.


Joly

On Fri, Jan 24, 2020 at 6:26 PM Ben Cannon > wrote:


I started what became 6x7 with a 64k ISDN line.   And 9600 baud modems…

in ’93 or so.  (I was a child, in Jr High…)

-Ben.


-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC
b...@6by7.net 




On Jan 24, 2020, at 3:21 PM, b...@theworld.com
 wrote:


On January 24, 2020 at 08:55 aar...@gvtc.com
 (Aaron Gould) wrote:

Thanks Jared, When I reminisce with my boss he reminds me that
this telco/ISP here initially started with a 56kbps internet
uplink , lol


Point of History:

When we, The World, first began allowing the general public onto the
internet in October 1989 we actually had a (mildly shared*) T1
(1.544mbps) UUNET link. So not so bad for the time. Dial-up customers
shared a handful of 2400bps modems, we still have them.

* It was also fanned out of our office to a handful of Boston-area
customers who had 56kbps or 9600bps leased lines, not many.

-- 
   -Barry Shein


Software Tool & Die    | b...@theworld.com
 | http://www.TheWorld.com

Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*




--
---
Joly MacFie 218 565 9365 Skype:punkcast
--
-

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Bryan Holloway]

On 1/27/20 1:42 PM, Aled Morris via NANOG wrote:
On Mon, 27 Jan 2020 at 12:13, Rob Pickering > wrote:


Wasn't the 56/64k thing a result of CAS (bit robbed) signalling
which was a fudge AT&T did to transport signalling information
in-band on T1s by stealing the low order bit for OOB signalling (it
wasnt actually every low order bit, but meant you had to throw away
every low order bit as CPE didn't know which ones were "corrupted"
by the carrier).
Proper ISDN was always 64kbit/s clear path with separate D channels
carried OOB end to end, away from the B channel data.


There was some element of interoperability required with the 
pre-existing data network architecture based on 56k channels and T1 
bearers.  This article has the detail:


https://en.wikipedia.org/wiki/T-carrier

/Soon after commercial success of T1 in 1962, the T1 engineering
team realized the mistake of having only one bit to serve the
increasing demand for housekeeping functions. They petitioned AT&T
management to change to 8-bit framing. This was flatly turned down
because it would make installed systems obsolete./


Compared to what was to follow, that all had to suffer the 56k channel 
limitation, there can't have been that many installed systems in 1962!


Aled



I seem to also recall that you couldn't use a 56k modem unless the 
far-end was digital.

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Bryan Holloway]

... and disabling call-waiting ... ;)


On 1/27/20 1:55 PM, John Von Essen wrote:


In those early days I remember setting up a download to start before bed 
so it could run all night, then wake up the morning to see my freshly 
downloaded 300KB file — assuming the phone line remained stable.


-John

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[Bryan Holloway]

Whoa. Gandalf.

I worked on one of those once and it was cray-zee. Customer bought 
one, and I had to get it to interoperate with an Ascend 400. It took a 
lot of fiddle-farting, but I did eventually get it to work.


Fun times.


On 1/27/20 8:00 PM, Jamie Bowden via NANOG wrote:

That was the other half of going to Extended Super Frame.  Lyle talked about 
AMI going away below, but didn't mention what replaced it (Binary 8bit Zero 
Substitution for the kids on the list).

I don't know about the other ILECs out there, but I don't know if Verizon will 
even provision a T1 anymore.  I know you can still get a PRI (that's how our 
phone systems interface with the PSTN), but if we needed a CT1 instead, I don't 
know that they'd be able (willing) to deliver it.  I know you can't get a BRI.  
We moved offices a few years ago and we basically lost the ability to use our 
STEs for anything but voice as we couldn't get BRIs delivered to the new space.

Speaking of ISDN, I had equipment that would support 56k ISDN, but never saw it 
provisioned (was that Switch56?  Or am I mixing up FR and ISDN?).  All of the 
ISDN circuits I dealt with were standard 2B+D (BRI), or 23B+D (PRI).  I think 
the oldest (and weirdest) piece of gear I personally worked on was a Gandalf 
ISDN router that was supporting a US Navy site to site connection.  Which makes 
me a newcomer to The Internet compared to a lot of people on this list, I'm 
sure.

Re: akamai yesterday - what in the world was that

[Bryan Holloway]

Is 10G enough? ;)

We just lit up several 100G Akamai links. Saved the day fo sho ... (this 
time.)



On 2/11/20 8:26 PM, Aaron Gould wrote:
Huge!  Big as ever.  My aanp links are (were) pegged, seriously.  I will 
be contacting Akamai about lighting up an additional 10 gig link to my 
local clusters.  Started at 12 noon central… still going pretty 
heavily.  Game/update release ?


-Aaron

*From:*Tom Deligiannis [mailto:tom.deligian...@gmail.com]
*Sent:* Tuesday, February 11, 2020 5:41 PM
*To:* aar...@gvtc.com
*Cc:* Nanog@nanog.org
*Subject:* Re: akamai yesterday - what in the world was that

There is a major update that has released today, how's everything 
looking for everyone?


Tom

Re: Chairman Pai Proposes Mandating STIR/SHAKEN To Combat Robocalls

[Bryan Holloway]

On 3/7/20 8:03 AM, Christopher Morrow wrote:

On Fri, Mar 6, 2020 at 11:05 PM Brian J. Murrell  wrote:


So, if my telco can bill the callers for those premium calls, they
surely know who they are, or at least know where they are sending the
bill and getting payment from.


You are mistaken, billing is very hard.
Telcos show this regularly.



On the contrary: billing is easy. Getting it right is hard.

Re: akamai yesterday - what in the world was that

[Bryan Holloway]

On 3/9/20 11:02 PM, Keith Medcalf wrote:


Warzone is a 83-101GB download for new, free-to-play users*.

And I remember the days when that would have taken 10 and a half years to 
download and consumed 56,000 floppy diskettes.

My, how times have changed!



"Never underestimate the bandwidth of a station-wagon full of tapes."

Re: akamai yesterday - what in the world was that

[Bryan Holloway]

We hit over 40G on one of our PNIs.

Currently, however, I'm trying to figure out why we're still seeing a 
significant amount of traffic over transit when we have PNIs at the same 
locations ...


I've reached out to Akamai, but I haven't heard anything back yet. I'm 
sure they're busy ...



On 3/10/20 10:14 PM, Aaron Gould wrote:
Wow, yeah, my Akamai servers are again, hitting all time highs… one 
cache hit up to ~30 gig… been ramping up and down since this morning 
around 9 or 10 a.m. central time.


Here’s a strange thing though, around 14:45 – 15:30, I got massive 
outbound on my internet connection (~20 gbps), and I never send that 
much out to the internet


-Aaron

AS27594 / UTSA contact?

[Bryan Holloway]

Howdy ... if anyone from University of Texas, San Antonio (AS27594) is 
lurking, could you please reach out to me off-list?


We have a mutual reachability problem through an IX in Dallas.

Thanks!

Re: IS-IS IPAM platform

[Bryan Holloway]

+1

On 4/13/20 4:02 PM, Tom Beecher wrote:

My recommendation would be not to bother. :)

Just encode the router loopback IPv4 address in the system identifier 
bytes and call it a day.


On Mon, Apr 13, 2020 at 9:55 AM JASON BOTHE via NANOG > wrote:


Does anyone have any recommendations for a database or IPAM platform
that can house IS-IS addressing?  Can’t seem to find anything out
there.

Thanks

J~

Re: AS27594 / UTSA contact?

[Bryan Holloway]

I'm good -- many thanks to those who reached out!


On 4/11/20 10:20 AM, Bryan Holloway wrote:
Howdy ... if anyone from University of Texas, San Antonio (AS27594) is 
lurking, could you please reach out to me off-list?


We have a mutual reachability problem through an IX in Dallas.

Thanks!

Re: IS-IS IPAM platform

[Bryan Holloway]

I've always wondered about folks' opinions about one thing, though:

In y'all's opinion, do you prefer/recommend using base-10 digits or hex 
in your NSAP addresses? I like the former for readability, but the 
latter can (could) be better for automation. Maybe.


I got into a heated argument about this once with ATM back in the day, 
but my brain's to frazzled to remember the takeaways.



On 4/13/20 7:37 PM, Randy Bush wrote:

Just encode the router loopback IPv4 address in the system identifier bytes
and call it a day.


i think asp wrote this up back in the early '90s.  anyone have a cite?

randy

Re: attribution

[Bryan Holloway]

On 4/13/20 10:31 PM, Randy Bush wrote:

I’m using CAIDA’s bgpreader and this one looks like it might be an
example of what you want.

R|R|1586714402.00|routeviews|route-views.eqix|||2914|206.126.236.12|103.148.41.0/24|206.126.236.12|2914
 58717 134371 134371 134371 134371 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 140076 
140076 140076 140076 140076 140076 140076 140076 140076 140076|140076|2914:410 
2914:1405 2914:2406 2914:3400||


aut-num:AS140076
as-name:MIS-AS-AP
descr:  Mir Internet Service
country:BD
org:ORG-MIS3-AP
admin-c:MISA2-AP
tech-c: MISA2-AP
mnt-by: APNIC-HM
mnt-irt:IRT-MIS-BD
mnt-routes: MAINT-MIS-BD
mnt-lower:  MAINT-MIS-BD
last-modified:  2020-01-31T06:35:38Z
source: APNIC

actually, an example of what none of us wants :)

it seems a lot of folk think prepending acrually works.

thanks



Oh, it works ... just not for anything pragmatically useful.

Re: mail admins?

[Bryan Holloway]

On 4/23/20 6:43 AM, John Osmon wrote:

On Wed, Apr 22, 2020 at 08:05:39AM +0300, Töma Gavrichenkov wrote:

On Wed, Apr 22, 2020, 12:45 AM Randy Bush  wrote:


sad.  http://nanog.org used to be the brilliant example of a fully
featured web site sans javascript, flash, ...



That was long ago now.  It was using Cvent for everything meeting-related
for 3 years already, and Cvent doesn't feel good with JS turned off.


Yes -- I think we all understand the technical problems with the site.

Thanks for helping Randy with being precise.



https://www.youtube.com/watch?v=tJ-LivK4-78

Sorry -- couldn't resist. Believe me, I've been on the receiving end of 
this myself.

Re: mail admins?

[Bryan Holloway]

On 4/24/20 4:58 PM, Michael Thomas wrote:


On 4/23/20 8:48 PM, Matt Palmer wrote:

On Thu, Apr 23, 2020 at 07:47:58PM -0700, Michael Thomas wrote:

On 4/23/20 7:35 PM, Matt Palmer wrote:
While I do think webauthn is a neat idea, and solves at least one 
very real
problem (credential theft via phishing), you do an absolutely 
terrible job

of making that case.

see RFC 4876, it is not about phishing. not even a little bit. Never has
been.
Whilst I do *absolutely* agree with you that "A Configuration Profile 
Schema
for Lightweight Directory Access Protocol (LDAP)-Based Agents" is "not 
about
phishing, not even a little bit", I'm not entirely sure how it 
advances your

argument.


sorry, 7486.

Mike



Shall we play a game?

https://en.wikipedia.org/wiki/Mastermind_(board_game)

finishline.com

[Bryan Holloway]

If anyone from finishline.com is lurking, could you please reach out to 
me off-line?


Our AS seems to be blocked to your web-site as a whole, which appears to 
be hosted by Akamai.


However, the curious thing is that we see this behavior from other IPs 
not part of our AS, so it's difficult to tell if it's just us, Akamai, 
or something more systemic.


E-mails to various contacts are going unanswered.

Thank you!

Re: Integrated WIFI router and phone adapter

[Bryan Holloway]

+1

But yes -- GPON.


On 5/18/20 9:03 AM, Mark Tinka wrote:



On 18/May/20 07:00, K MEKKAOUI wrote:


Hi NANOG Community

Anyone knows about a good integrated WIFI router and phone adapter 
that can be used to provide home and business internet and phone 
service. We tried couple of them but we’ve seen some instability and 
reliability issues (i.e. wifi issues, phone issues, etc.). Also some 
of them are designed to work better over DSL but not over DOCSIS.




Have you looked at Calix:

https://www.calix.com/platforms/non-exos-premises-systems/gigafamily-overview/gigahubs.html

You don't mention what last mile you're using. The Calix Giga units are 
for GPON.


Mark.

Rate-limiting BCOP?

[Bryan Holloway]

I'm curious if the community would be willing to share their 
best-practices and/or recommendations and thoughts on how they handle 
situations where a customer buys X amount of bandwidth, but the physical 
link is capable of Y, where Y > X. (Yes, I speak of policy-maps, 
tx/rx-queues, etc.)


For example, it might be arguably common to aggregate customer links 
Layer 2, and then push them upstream to where they anchor Layer 3. That 
Layer 2 <-> Layer 3 could be a couple of meters or several kilometers.


So, as I see it, my options are:

* Rate-limit at the Layer 2 switch for both customer ingress/egress,

* Rate-limit at the Layer 3 router upstream, i/e, or

* Some combination thereof? E.g.: Rate-limit my traffic towards the 
customer closer to the core, and rate-limit ingress closer to the edge?


I've done all three on some level in my travels, but in the past it's 
also been oftentimes vendor-centric which hindered a scalable or 
"templateable" solution. (Some things police in only one direction, or 
only well in one direction, etc.)


In case someone is interested in a tangible example, imagine an Arista 
switch and an ASR9k router. :)


Thoughts?

Re: IPv4 Broker / Service -

[Bryan Holloway]

+1

On 6/11/20 9:38 PM, b...@theworld.com wrote:


Addrex.net

I know some of the principles personally and would vouch for them.


On June 11, 2020 at 14:27 edwin.malle...@gmail.com (edwin.malle...@gmail.com) 
wrote:
  > Hi Nanog,
  >
  >
  >
  > I have need of a reputable IPv4 broker or service  ? personal experience 
with
  > said broker would be preferred.  These would be for small blocks - /23, 24s 
?
  > in the US, so ARIN.  I know, I know, IPv6 for life and all that and I agree,
  > but ? you know, the business.  I?m happy to take responses off-list, but I
  > would really appreciate any recommendations.
  >
  >
  >
  > Thanks!
  >
  >
  >
  > Ed
  >

Microsoft AS8075 contact?

[Bryan Holloway]

Hello ...

If anyone from Microsoft peering is lurking, I could use an assist.

We have a reachability issue in Chicago.

E-mail to their PeeringDB NOC contact have gone unanswered.

Thank you!

Re: Microsoft AS8075 contact?

[Bryan Holloway]

I'm in good hands ... thanks to all who responded.


On 6/18/20 2:33 PM, Bryan Holloway wrote:

Hello ...

If anyone from Microsoft peering is lurking, I could use an assist.

We have a reachability issue in Chicago.

E-mail to their PeeringDB NOC contact have gone unanswered.

Thank you!

atmark trading

[Bryan Holloway]

Tired of receiving spam from these jamokes.

https://www.atmarktrade.com/

Atmark Trading out of Chicago.

Have tried unsubscribing numerous times; e-mailed their "info" accounts 
to no avail.


My only recourse, now, is to shame.

Doubt it will do any good, but if anyone has a contact who can actually 
get us off their ridiculous mailing-list -- 3 to 4 e-mails a day --, 
well, I'm all ears.


If not, well, don't do business with these bozos.

/rant

Happy Weekend ...

Re: atmark trading

[Bryan Holloway]

It's not sales; it's some dumb mailing list managed by "Soundest", which 
is now owned by "Omnisend", which sounds even less fun than its predecessor.


Atmark's web-site has no contacts or management information listed other 
than "info@", otherwise I would do what you suggest.


I don't have the patience to call their 800 number and talk to someone 
who has zero interest in getting me off of their mailing-list, assuming 
the drone has even an inkling of what I'm talking about.



On 8/22/20 10:44 PM, Mike Hale wrote:
I've found it useful to email management if certain sales people refuse 
to stop contacting you.


On Sat, Aug 22, 2020, 1:34 PM Bryan Holloway <mailto:br...@shout.net>> wrote:


Tired of receiving spam from these jamokes.

https://www.atmarktrade.com/

Atmark Trading out of Chicago.

Have tried unsubscribing numerous times; e-mailed their "info" accounts
to no avail.

My only recourse, now, is to shame.

Doubt it will do any good, but if anyone has a contact who can actually
get us off their ridiculous mailing-list -- 3 to 4 e-mails a day --,
well, I'm all ears.

If not, well, don't do business with these bozos.

/rant

Happy Weekend ...

Re: atmark trading

[Bryan Holloway]

On 8/22/20 11:06 PM, Eric Tykwinski wrote:



On Aug 22, 2020, at 4:53 PM, Bryan Holloway  wrote:

It's not sales; it's some dumb mailing list managed by "Soundest", which is now owned by 
"Omnisend", which sounds even less fun than its predecessor.

Atmark's web-site has no contacts or management information listed other than 
"info@", otherwise I would do what you suggest.

I don't have the patience to call their 800 number and talk to someone who has 
zero interest in getting me off of their mailing-list, assuming the drone has 
even an inkling of what I'm talking about.



Dumb question, but if it’s a mailman or similiar list does it have unsubscribe 
headers?
List-Unsubscribe: <https://mailman.nanog.org/mailman/options/nanog>
List-Unsubscribe: <mailto:nanog-requ...@nanog.org?subject=unsubscribe>
List-Subscribe: <https://mailman.nanog.org/mailman/listinfo/nanog>, 
<mailto:nanog-requ...@nanog.org?subject=subscribe>

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300



There's a link to "unsubscribe", which I've done several times. No joy.

Re: atmark trading

[Bryan Holloway]

Indeed, Mel, and I agree wholeheartedly.

I suppose it's the optimist in me that people should behave according to 
their supposed attempts to allow things like, oh, I dunno ... 
"unsubscribe" links that actually do what they claim to do?


Meh. I'm done ranting ...

P.S.: Don't do business with Atmark.

Whe! Onwards.

EOThread


On 8/22/20 11:21 PM, Mel Beckman wrote:

Bryan,

This is what inbound mail filters are for. Regex them to oblivion and get on 
with your life. No reason to waste another millisecond on them.

  -mel


On Aug 22, 2020, at 2:14 PM, Bryan Holloway  wrote:



On 8/22/20 11:06 PM, Eric Tykwinski wrote:

On Aug 22, 2020, at 4:53 PM, Bryan Holloway  wrote:


It's not sales; it's some dumb mailing list managed by "Soundest", which is now owned by 
"Omnisend", which sounds even less fun than its predecessor.

Atmark's web-site has no contacts or management information listed other than 
"info@", otherwise I would do what you suggest.

I don't have the patience to call their 800 number and talk to someone who has 
zero interest in getting me off of their mailing-list, assuming the drone has 
even an inkling of what I'm talking about.


Dumb question, but if it’s a mailman or similiar list does it have unsubscribe 
headers?
List-Unsubscribe: <https://mailman.nanog.org/mailman/options/nanog>
List-Unsubscribe: <mailto:nanog-requ...@nanog.org?subject=unsubscribe>
List-Subscribe: <https://mailman.nanog.org/mailman/listinfo/nanog>, 
<mailto:nanog-requ...@nanog.org?subject=subscribe>
Sincerely,
Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300


There's a link to "unsubscribe", which I've done several times. No joy.

Re: Centurylink having a bad morning?

[Bryan Holloway]

Not everyone will peer with you, notably, AS3356 (unless you're big 
enough, which few can say.)


On 8/31/20 4:33 PM, Tomas Lynch wrote:
Maybe we are idealizing these so-called tier-1 carriers and we, tier-ns, 
should treat them as what they really are: another AS. Accept that they 
are going to fail and do our best to mitigate the impact on our own 
networks, i.e. more peering.


On Mon, Aug 31, 2020 at 9:54 AM Martijn Schmidt via NANOG 
mailto:nanog@nanog.org>> wrote:


At this point you don't even know whether it's a human error
(example: generating a flowspec rule for port TCP/179), a filtering
issue (example: accepting a flowspec rule for port TCP/179), or a
software issue (example: certain flowspec update crashes the BGP
daemon). And in the third scenario I think that at least some
portion of the blame shifts from the carrier to its vendors,
assuming the thing that crashed was not a home-grown BGP implementation.

With the route optimizer incidents - because let's face it, Honest
Networker is on the money as usual
https://honestnetworker.net/2020/08/06/as10990-routing/ - there is
really no excuse for any tier-1 carrier, they should at the very
least have strict prefix-list based filtering in place for
customer-facing EBGP sessions. In those cases it's much easier to
state who's not taking care of their proverbial lawn.

Best regards,
Martijn

On 8/31/20 3:25 PM, Tom Beecher wrote:



https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/


I definitely found Mr. Prince's writing about yesterday's events
fascinating.

Verizon makes a mistake with BGP filters that allows a secondary
mistake from leaked "optimizer" routes to propagate, and Mr.
Prince takes every opportunity to lob large chunks of granite
about how terrible they are.

L3 allows an erroneous flowspec announcement to cause massive
global connectivity issues, and Mr. Prince shrugs and says
"Incidents happen."





On Mon, Aug 31, 2020 at 1:15 AM Hank Nussbacher
mailto:h...@interall.co.il>> wrote:

On 30/08/2020 20:08, Baldur Norddahl wrote:


https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/

Sounds like Flowspec possibly blocking tcp/179 might be the cause.

But that is Cloudflare speculation.

Regards,
Hank
Caveat: The views expressed above are solely my own and do not
express the views or opinions of my employer


An outage is what it is. I am not worried about outages. We
have multiple transits to deal with that.

It is the keep announcing prefixes after withdrawal from
peers and customers that is the huge problem here. That is
killing all the effort and money I put into having
redundancy. It is sabotage of my network after I cut the
ties. I do not want to be a customer at an outlet who has a
system that will do that. Luckily we do not currently have a
contract and now they will have to convince me it is safe for
me to make a contract with them. If that is impossible I
guess I won't be getting a contract with them.

But I disagree in that it would be impossible. They need to
make a good report telling exactly what went wrong and how
they changed the design, so something like this can not
happen again. The basic design of BGP is such that this
should not happen easily if at all. They did something
unwise. Did they make a route reflector based on a database
or something?

Regards,

Baldur

On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho
mailto:mikeboli...@gmail.com>> wrote:

Exactly. And asking that they somehow prove this won't
happen again is impossible.

- Mike Bolitho

On Sun, Aug 30, 2020, 8:10 AM Drew Weaver
mailto:drew.wea...@thenap.com>>
wrote:

I’m not defending them but I am sure it isn’t
intentional.

*From:* NANOG
mailto:thenap@nanog.org>> *On Behalf Of *Baldur
Norddahl
*Sent:* Sunday, August 30, 2020 9:28 AM
*To:* nanog@nanog.org 
*Subject:* Re: Centurylink having a bad morning?

How is that acceptable behaviour? I shall remember
never to make a contract with these guys until they
can prove that they won't advertise my prefixes after
I pull them. Under any circumstances.

søn. 30. aug. 2020 15.14 skrev Joseph Jenkins
mailto:j...@breathe-underwater.com>>:

Finally got through on their support line and
spoke to level1. The only thing the tech could
say was

Re: Centurylink having a bad morning?

[Bryan Holloway]

On 9/2/20 1:49 PM, Nick Hilliard wrote:

Shawn L via NANOG wrote on 02/09/2020 12:15:

We once moved a 3u server 30 miles between data centers this way.
Plug redundant psu into a ups and 2 people carried it out and put
them in a vehicle.


hopefully none of these server moves that people have been talking about 
involved spinning disks.  If they did, kit damage is one of the likely 
outcomes - you seriously do not want to bump active spindles:


www.google.com/search?q=disk+platter+damage&tbm=isch

SSDs are a different story. In that case it's just a bit odd as to why 
you wouldn't want to power down a system to physically move it - in the 
sense that if your service delivery model can't withstand periodic 
maintenance and loss of availability of individual components, 
rethinking the model might be productive.


Nick



If it's your server, moving beyond (very) local facilities, and time is 
not of the essence, then sure: power down.


If you're law-enforcement mid-raid, or trying to preserve your Frogger 
high-score, well, ...

Re: The great Netflix vpn debacle!

[Bryan Holloway]

So I've made some progress, but not on the HBO front. (Hulu and Netflix 
have been responsive so far.)


Tried the e-mail address on Mike Hammett and Co.'s handy web-page, but 
got no response after several days. Ironically we were able to get 
through to the "closed-captioning" department, but this isn't 
particularly useful.


Does anyone have another possible contact for HBO folks to get some 
prefixes unflagged as "VPN"?


To be clear, this is not a geolocate issue. At least according to the 
error our users are getting.


Thanks, all!


On 8/28/21 1:51 AM, Justin Krejci wrote:

+1 on Bryan's message.


TL;DR

It seems lots of ISPs are struggling to figure out the why and the where 
of many IP addresses or blocks that are suddenly being blacklisted or 
flagged as VPNs or as out of service area.





I would really love to find, as Bryan said, if there is one particular 
IP reputation data provider who either got real aggressive recently or 
some (contaminated?) data was shared around. If there is I have no 
problem wading through their support processes to get it sorted but as 
it stands I just don't know who to call. It just has been very difficult 
to glean anyactionable info and of course the normal support teams at 
the respective streaming providers mostly just are telling customers to 
call their ISP as if every random ISP has some special backdoor 
contact to every streaming provider where we can just get problems 
resolved quickly and easily while we all have a good laugh at people 
being able to watch their preferred movies and shows.



At least with email DNSBL filtering you usually get informed which DNSBL 
you are listed on and you can sort that out directly. In this case, the 
overall system of IP reputation based filtering seems still 
comparatively immature. The most I have gotten is after a very long 
phone call with someone at Hulu, they confirmed there is some issue 
affecting multiple networks and they are working on the issue and 
suggested I go through a whitelisting request process which may solve 
the problems but just for Hulu obviously.



I have published and tried to register our own geofeed data as defined 
in RFC8805 with as many IP geolocation providers as possible. I have 
checked around to as many IP geolocation and IP reputations sites as I 
can find and everything is either clean/accurate or there is no query 
method open to the public for troubleshooting that I can find. This is 
just yet another example to me of immaturity on dealing with geolocation 
problems: just spinning my wheels in the dark with mud spraying 
everywhere. There does not appear to be any consistency on handling 
issues by the content providers using IP geolocation and reputation to 
filter. If the content providers want to reject client connections they 
ought to provide more actionable information in their errors messages 
for ISPs since they are all just telling the users to call their ISPs. 
It just feels like a vicious circle.



So currently we are left with multiple video streaming providers that 
all started to flag many customers across many of our IP blocks all 
beginning earlier this month affecting customers, many of whom have been 
using the same IP address for years without issue until now. Do we try 
and decommission multiple IP subnets shuffle users over to new subnets 
and risk contaminating more subnets if this is an ongoing and 
regularly updated blacklist data set. This would further exacerbate the 
problem across yet more subnets that are getting scarcer. As a tangent, 
I am curious to see how IP geolocation and reputation systems are 
handling IPv6, I suppose they are just grouping larger and larger 
networks together into the same listings.



Someone who knows something concrete about this current issue, please 
throw us ISPs a bone.



With this email I feel like Leia recording a video plea for help 
addressed to Obi-Wan Kenobi help me Nanog Community... you're my 
only hope.






----
*From:* NANOG  on behalf 
of Bryan Holloway 

*Sent:* Friday, August 27, 2021 4:56 PM
*To:* Mike Hammett; John Alcock
*Cc:* nanog@nanog.org
*Subject:* Re: The great Netflix vpn debacle!
Is there some new DB that major CDNs are using?

We've been getting several reports of prefixes of ours being blocked,
claiming to be VPNs, even though we've been using those subnets without
incident for years.

HBO, Netflix, and Hulu appear to be common denominators. I have to
wonder if they're all siphoning misinformation off of some new DB
somewhere ...


On 8/14/21 1:45 AM, Mike Hammett wrote:
https://thebrotherswisp.com/index.php/geo-and-vpn/ 

<https://thebrotherswisp.com/index.php/geo-and-vpn/>




-
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/ 
<http://www.ics-il.com/>>
<*MailScanner has detected a possible 

Re: The great Netflix vpn debacle!

[Bryan Holloway]

Indeed.

Let me be 100% clear:

We are having issues with prefixes flagged as VPNs. They are not.

We are NOT having issues with prefixes and geolocation.



On 8/31/21 9:24 PM, Niels Bakker wrote:

* war...@kumari.net (Warren Kumari) [Tue 31 Aug 2021, 21:04 CEST]:
So, RFC8805 is great and all, but it sure is annoying that you have to 
find

webforms for a whole heap-o-geolocation providers, and figure out how to
tell them where your geofeed file lives, etc.

Introducing RFC9092 - "Finding and Using Geofeed Data" (

[..]

This won't help at all against geolocation vendors marking proxies and 
VPN endpoints as such.



 -- Niels.

Re: The great Netflix vpn debacle!

[Bryan Holloway]

Thanks, Owen ... good point.

Now hearing reports for these same prefixes with Disney+ too.

So the common denominators are:

HBO
Hulu
Netflix
Amazon Prime
Disney+

... there has _got_ to be some new-fangled DB somewhere. This all 
started in the last month or so.


All of our RR objects, whois, DNS is solid ... dehr?

Fun times.


On 8/31/21 9:16 PM, Owen DeLong wrote:

[snip]



Geolocate and VPN or Not are often kind of tied to the same kinds of reporting 
services and it may well be that whatever provider HBO is using for one is also 
being used for the other.

Owen

Fastly Peering Contact?

[Bryan Holloway]

Hey all ... looking for a Fastly (54113) peering contact that might be 
able to get me in touch with the right folks to do stuff.


E-mails to the 'policy' peeringdb contact don't seem to be getting through.

Thanks!

- bryan

Anyone from Cloudflare peering about?

[Bryan Holloway]

Trying to fix a peer and not getting much traction through peering@ ...

Feel free to contact me off-list. Thanks!

Re: Anyone from Cloudflare peering about?

[Bryan Holloway]

Heard from not one, not two, but THREE folks at Cloudflare!

Now that's service.

Thanks, everyone!


On 8/21/22 13:11, Bryan Holloway wrote:

Trying to fix a peer and not getting much traction through peering@ ...

Feel free to contact me off-list. Thanks!

AMS-IX @ 350 Cermak issue?

[Bryan Holloway]

Anyone else having issues with AMS-IX at 350 Cermak?

Our circuit has been bouncing on and off all day. I'd suspect our 
optics/port, except that when it's up, some BGP sessions come up, but 
not all. The ones that don't come up aren't pingable.


Leads me to believe it's not us.

E-mails to their NOC have gone unanswered.

Thanks all ...

Anyone from Verisign peering lurking?

[Bryan Holloway]

Trying to resolve a peering issue; e-mails to listed contacts have been 
fruitless ... thanks!

Comcast circuit guru lurking?

[Bryan Holloway]

Looking for some help or direction with MTU issues on a recently 
installed point-to-point circuit.


Attempts to rectify have involved front-line PMs/folks who don't grok 
the problem.


Please contact me off-list, thank you!!

- bryan

Looking for an AS60068 (CDN77) contact ...

[Bryan Holloway]

... in the peering department.

If there's anyone lurking, could you please contact me off-list? Thanks!

- bryan

Re: Best Linux (or BSD) hosted BGP?

[Bryan Holloway]

Curious to hear more specifics about your IS-IS assertion.

We've been running it on FRR for some time without incident, but I'll 
concede that we don't do very much with it other than saying, "hey -- 
we're here; oh, and you're there."



On 5/4/23 06:04, Mark Tinka wrote:



On 5/4/23 00:51, Matt Corallo wrote:

Lots of replies saying which of BIRD/exabgp/frr/quagga/openbgpd folks 
prefer, but they're all pretty good. Honestly for such a project 
they're all just as great, it comes down mostly to what you're used to 
config-wise. Used to big metal router configuration? You might find 
BIRD foreign. Used to more functional code stuff? BIRD is pretty 
great. Others I have less experience with.


IS-IS in Quagga and FRR are not yet ready for business, is what I would 
caution.


I don't know if the other options support it or not.

Mark.

Re: Best Linux (or BSD) hosted BGP?

[Bryan Holloway]

On 5/8/23 07:03, Mark Tinka wrote:



On 5/8/23 00:22, Bryan Holloway wrote:


Curious to hear more specifics about your IS-IS assertion.

We've been running it on FRR for some time without incident, but I'll 
concede that we don't do very much with it other than saying, "hey -- 
we're here; oh, and you're there."


Talking to other vendors, or other FRR installations?

Mark.


You said, "IS-IS in Quagga and FRR are not yet ready for business, ..."

Not ready for business in what way? Performance? Cross-vendor 
compatibility? Features?


Or did I misunderstand your statement?

Re: Best Linux (or BSD) hosted BGP?

[Bryan Holloway]

On 5/8/23 18:45, Mark Tinka wrote:



On 5/8/23 15:44, Bryan Holloway wrote:


You said, "IS-IS in Quagga and FRR are not yet ready for business, ..."

Not ready for business in what way? Performance? Cross-vendor 
compatibility? Features?


Or did I misunderstand your statement?


Broken when talking to Cisco IOS XE. Catalogued here:

https://lists.frrouting.org/pipermail/frog/2023-March/001265.html

I have no doubt FRR can talk IS-IS to other instances of FRR, but that 
is not a realistic scenario in a large scale network with multiple vendors.


Mark.



Interesting, and thank you.

FWIW, we're running it against regular IOS, IOS-XR, and Arista with no 
issues (so far ...)

TACACS+ server recommendations?

[Bryan Holloway]

Ah, the good old days when I could download the latest tac_plus code 
from the Cisco FTP site, compile it, and off I go.


But I digress.

Curious if there are any operators out there that have a good 
recommendation on a lightweight TACACS+ server for ~200 NEs and 
access-control for 20-30 folks. Nothing too special, but some sort of 
simple command-control auth would be nice.


Open-source is fine ... we've been looking at commercial options, but 
they're all pretty pricey and have way more features than we need.


Thank you all in advance!

- bryan

Re: SMTP-friendly VPS provider where I can also get a BGP feed

[Bryan Holloway]

Not sure if this helps, but they only appear to block 25 for IPv4.

IPv6 works fine.

Supposedly you can open a support-ticket to have this block removed, but 
I'm assuming you've already done that?


- bryan


On 9/26/23 12:09, Daniel Corbe wrote:

Hey all,

I apologize if this isn't the right place to post this; however, I 
thought maybe the NANOG community would be able to point me in the right 
direction.


I'm looking for a place that I can host a mailer.  My primary use case 
is a Mailman-style technical discussion list; much like NANOG but 
software related instead of network related: READ: non-commercial in 
nature.


I'm currently a vultr customer, but they're refusing to unblock port 25 
on my account.  I've tried explaining my use case but no matter who I 
talk to over there they just keep pointing me to their spam policy.


Thanks!
-Daniel

Re: FB?

[Bryan Holloway]

On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the 
product), much love to all you FB engineers that went through (and are 
probably still going through) much hell.




+1 on both counts.

We've all been there; no bueno.

Re: Help on setting up a new block

[Bryan Holloway]

On 3/20/19 10:28 AM, John Alcock wrote:
I found an interesting pattern.  I see a lot of traffic stopping at 
softlayer.com .  Big datacenter?  Could they be 
doing some blocking?


John



Could be. They were acquired by IBM a few years ago.

Re: well-known Anycast prefixes

[Bryan Holloway]

On 3/21/19 10:59 AM, Frank Habicht wrote:

Hi James,

On 20/03/2019 21:05, James Shank wrote:

I'm not clear on the use cases, though.  What are the imagined use cases?

It might make sense to solve 'a method to request hot potato routing'
as a separate problem.  (Along the lines of Damian's point.)


my personal reason/motivation is this:
Years ago I noticed that my traffic to the "I" DNS root server was
traversing 4 continents. That's from Tanzania, East Africa.
Not having a local instance (back then), we naturally sent the traffic
to an upstream. That upstream happens to be in that club of those who
don't have transit providers (which probably doesn't really matter, but
means a "global" network).


/snip


Greetings,
Frank



I can think of another ...

We rate-limit DNS from unknown quantities for reasons that should be 
obvious. We white-list traffic from known trusted (anycast) ones to 
prevent a DDoS attack from throttling legitimate queries. This would be 
a useful way to help auto-generate those ACLs.

Re: well-known Anycast prefixes

[Bryan Holloway]

On 3/21/19 11:52 AM, Ross Tajvar wrote:
Not all any-casted prefixes are DNS resolvers and not all DNS resolvers 
are anycasted. It sounds like you would be better served by a list of 
well-known DNS resolvers.


True on both counts, and that's why I said "help".


On Thu, Mar 21, 2019 at 12:35 PM Bryan Holloway <mailto:br...@shout.net>> wrote:



On 3/21/19 10:59 AM, Frank Habicht wrote:
 > Hi James,
 >
 > On 20/03/2019 21:05, James Shank wrote:
 >> I'm not clear on the use cases, though.  What are the imagined
use cases?
 >>
 >> It might make sense to solve 'a method to request hot potato
routing'
 >> as a separate problem.  (Along the lines of Damian's point.)
 >
 > my personal reason/motivation is this:
 > Years ago I noticed that my traffic to the "I" DNS root server was
 > traversing 4 continents. That's from Tanzania, East Africa.
 > Not having a local instance (back then), we naturally sent the
traffic
 > to an upstream. That upstream happens to be in that club of those who
 > don't have transit providers (which probably doesn't really
matter, but
 > means a "global" network).

/snip

 > Greetings,
 > Frank
 >

I can think of another ...

We rate-limit DNS from unknown quantities for reasons that should be
obvious. We white-list traffic from known trusted (anycast) ones to
prevent a DDoS attack from throttling legitimate queries. This would be
a useful way to help auto-generate those ACLs.

Re: Incoming SSDP UDP 1900 filtering

[Bryan Holloway]

On 3/25/19 9:08 AM, Tom Beecher wrote:
If your edge ingress ACLs are not 100% in sync all the time, you will 
inevitably have Really Weird Stuff happen that will end up taking 
forever to diagnose.


You will eventually end up closing off a port that something else needs 
to work properly, and now you have to figure out how to resolve that.


Packet filtering is more computationally taxing than just routing is. 
Your edge equipment is likely going to be built for maximum routing 
efficiency. Trying to bite off too much filtering there increases your 
risk of legit traffic being tossed on the floor.



Not necessarily disagreeing with your posits here, but, empirically 
speaking, we've had ACLs for stuff like this for years without any 
incidents or consternation.


And we are careful to ensure that any updates are pushed to all edge 
ingresses.




On Mon, Mar 25, 2019 at 6:41 AM Tom Hill > wrote:


On 25/03/2019 09:17, Sean Donelan wrote:
 > Its always a bad idea to do packet filtering at your bgp border.


Wild assertion. Why?

DoS mitigation, iACLs, BGP security... I can think of lots of very
sensible reasons.

-- 
Tom

Re: Did IPv6 between HE and Google ever get resolved?

[Bryan Holloway]

On 3/31/19 8:21 PM, Valdis Klētnieks wrote:

On Sun, 31 Mar 2019 18:10:09 -0700, Christopher Morrow said:


Apologies, I do actually see a path from 174 -> 6939 (well 28 paths):
   174  6939 

it's clearly not all of HE -> Cogent, and it's clearly not supposed to
be working (I would think).


Wait, what?

Are you saying that they refused to peer - and then failed at refusing? :)



Let them eat cake.

Re: Frontier rural FIOS & IPv6

[Bryan Holloway]

Furthermore, NAT, prevalent with IPv4, adds latency. There is none with 
IPv6 (unless you're doing it wrong.)



On 3/31/19 8:42 PM, Mike Leber wrote:

You are assuming the routing and transit relationships in IPv4 are the
same in IPv6.

IPv4 has many many many suboptimal transit relationships where routing
is purposely suboptimal on the part of the networks in the path due to
competitive reasons.  One example of suboptimal routing is traffic not
being exchanged in a closer location where both networks exist and
instead being routed hundreds or thousands of miles out of the way.

Customers don't get to influence the decisions of monopolies etc.

Customers choose based on inertia, brand experience, and what options
are even available to them to get IPv6 vs IPv4.

IPv6 has randomized some of these vendor relationships due to some
upstream networks not even implementing IPv6, meaning the downstream
networks were forced to make other choices.


On 3/31/19 6:21 PM, Keith Medcalf wrote:

It is not possible for web pages to load faster over IPv6 than over IPv4.  All other 
factors being equal, IPv6 has higher overhead than IPv4 for the same payload throughput.  
This means that it is physically impossible for IPv6 to be move payload bytes 
"faster" than IPv4 can move the same payload.

In other words, IPv6 has a higher "packet tax" than IPv4.  Since you have no choice but 
to pay the "packet tax" the actual payload data flows more slowly.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ca By
Sent: Sunday, 31 March, 2019 18:53
To: Matt Hoppes
Cc: Aaron C. de Bruyn; NANOG mailing list
Subject: Re: Frontier rural FIOS & IPv6



On Sun, Mar 31, 2019 at 4:20 PM Matt Hoppes
 wrote:


Going to play devils advocate.

If frontier has a ton of ipv4 addresses, what benefit is there
to them in rolling out ipv6?

What benefit is there to you?


I love xbox and xbox works better on ipv6,

https://www.nanog.org/sites/default/files/wed.general.palmer.xbox_.47
.pdf

Also, webpages load faster , and i love fast web pages

https://code.fb.com/networking-traffic/ipv6-it-s-time-to-get-on-
board/


https://www.akamai.com/fr/fr/multimedia/documents/technical-
publication/a-case-for-faster-mobile-web-in-cellular-ipv6-
networks.pdf




On Mar 31, 2019, at 7:11 PM, C. A. Fillekes
 wrote:




Still it's pretty darn good having real broadband on the
farm.  One thing at a time.


But, let's start thinking about ways to get Frontier up to
speed on the IPv6 thing.



On Sun, Mar 31, 2019 at 4:24 PM Aaron C. de Bruyn
 wrote:


You're not alone.

I talked with my local provider about 4 years ago and
they said "We will probably start looking into IPv6 next year".
I talked with them last month and they said "Yeah,
everyone seems to be offering it.  I guess I'll have to start reading
how to implement it".

I'm sure 2045 will finally be the year of IPv6
everywhere.

-A

On Sat, Mar 30, 2019 at 7:36 AM C. A. Fillekes
 wrote:



So by COB yesterday we now officially have FIOS
at our farm.


Went from 3Mbps to around 30 measured average.
Yay.


It's a business account, Frontier.  But...still
no IPv6.


The new router's capable of it.  What's the hold
up?


Customer service's response is "We don't offer
that".

Re: Frontier rural FIOS & IPv6

[Bryan Holloway]

I remember tapping the switch-hook to emulate pulse-dialing on 
touch-tone phones.


Few were impressed.


On 3/31/19 9:01 PM, Luke Guillory wrote:

My mom was cheap and only had pulse dialing in the 90s, it made using pagers 
difficult. Had to flip to tone after it dialed.



Ns

Sent from my iPad







On Mar 31, 2019, at 8:53 PM, Matt Hoppes  
wrote:


The telephone example:
What IS the benefit of DTMF other than I can dial faster?  None. And I can use 
IVRs. Again - no impact to me as a telephone company.

As far as ipv6. It’s been proven things “load faster” because the ipv6 servers 
of the various websites are not as heavily loaded as the ipv4 variants.

All things equal - ipv6 doesn’t load faster. There’s literally no advantage to 
ipv6 other than “I’m out of ipv4 and need to continue to provide public 
routable Ips to my customers. “


On Mar 31, 2019, at 9:42 PM, Mike Leber  wrote:

You are assuming the routing and transit relationships in IPv4 are the
same in IPv6.

IPv4 has many many many suboptimal transit relationships where routing
is purposely suboptimal on the part of the networks in the path due to
competitive reasons.  One example of suboptimal routing is traffic not
being exchanged in a closer location where both networks exist and
instead being routed hundreds or thousands of miles out of the way.

Customers don't get to influence the decisions of monopolies etc.

Customers choose based on inertia, brand experience, and what options
are even available to them to get IPv6 vs IPv4.

IPv6 has randomized some of these vendor relationships due to some
upstream networks not even implementing IPv6, meaning the downstream
networks were forced to make other choices.



On 3/31/19 6:21 PM, Keith Medcalf wrote:
It is not possible for web pages to load faster over IPv6 than over IPv4.  All other 
factors being equal, IPv6 has higher overhead than IPv4 for the same payload throughput.  
This means that it is physically impossible for IPv6 to be move payload bytes 
"faster" than IPv4 can move the same payload.

In other words, IPv6 has a higher "packet tax" than IPv4.  Since you have no choice but 
to pay the "packet tax" the actual payload data flows more slowly.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ca By
Sent: Sunday, 31 March, 2019 18:53
To: Matt Hoppes
Cc: Aaron C. de Bruyn; NANOG mailing list
Subject: Re: Frontier rural FIOS & IPv6



On Sun, Mar 31, 2019 at 4:20 PM Matt Hoppes
 wrote:


   Going to play devils advocate.

   If frontier has a ton of ipv4 addresses, what benefit is there
to them in rolling out ipv6?

   What benefit is there to you?


I love xbox and xbox works better on ipv6,

https://www.nanog.org/sites/default/files/wed.general.palmer.xbox_.47
.pdf

Also, webpages load faster , and i love fast web pages

https://code.fb.com/networking-traffic/ipv6-it-s-time-to-get-on-
board/


https://www.akamai.com/fr/fr/multimedia/documents/technical-
publication/a-case-for-faster-mobile-web-in-cellular-ipv6-
networks.pdf




   On Mar 31, 2019, at 7:11 PM, C. A. Fillekes
 wrote:




   Still it's pretty darn good having real broadband on the
farm.  One thing at a time.


   But, let's start thinking about ways to get Frontier up to
speed on the IPv6 thing.



   On Sun, Mar 31, 2019 at 4:24 PM Aaron C. de Bruyn
 wrote:


   You're not alone.

   I talked with my local provider about 4 years ago and
they said "We will probably start looking into IPv6 next year".
   I talked with them last month and they said "Yeah,
everyone seems to be offering it.  I guess I'll have to start reading
how to implement it".

   I'm sure 2045 will finally be the year of IPv6
everywhere.

   -A

   On Sat, Mar 30, 2019 at 7:36 AM C. A. Fillekes
 wrote:



   So by COB yesterday we now officially have FIOS
at our farm.


   Went from 3Mbps to around 30 measured average.
Yay.


   It's a business account, Frontier.  But...still
no IPv6.


   The new router's capable of it.  What's the hold
up?


   Customer service's response is "We don't offer
that".

Re: Disney+ CDN

[Bryan Holloway]

On 4/12/19 2:31 PM, Chris Grundemann wrote:




On Fri, Apr 12, 2019 at 3:03 PM Jared Geiger > wrote:


An article mentioned BAMTech's platform which is what NHL, MLB, and
HBO GO are built on. The bits from the first two come from Akamai
and Level3 CDNs. I haven't looked into where HBO Go comes from.


Yep, they decided to buy BAMTech and build their own:
https://www.thewaltdisneycompany.com/walt-disney-company-acquire-majority-ownership-bamtech/ 



So, on a practical level, with whom should I peer so as not to jack up 
my transit costs?

Re: Disney+ CDN

[Bryan Holloway]

On 4/26/19 4:33 PM, Ross Tajvar wrote:
Looks like Disney has an ASN for their streaming service: 
https://www.peeringdb.com/net/15627




Helluva entry ...

*crickets*
*tumbleweeds*

Re: looking for hostname router identifier validation

[Bryan Holloway]

On 4/29/19 3:13 PM, Eric Kuhnke wrote:
I would caution against putting much faith in the validity of 
geolocation or site ID by reverse DNS PTR records. There are a vast 
number of unmaintained, ancient, stale, erroneous or wildly wrong PTR 
records out there. I can name at least a half dozen ISPs that have 
absorbed other ASes, some of those which also acquired other ASes 
earlier in their history, forming a turducken of obsolete PTR records 
that has things with ISP domain names last in use in the year 2002.


I still see references to UUNet in some reverse PTRs.

So, uh, yeah.

Re: looking for hostname router identifier validation

[Bryan Holloway]

On 4/30/19 7:12 AM, Jared Mauch wrote:
While at NTT and at Akamai we have managed to publish sane PTR records 
and make the forward work as well. You need to automate it by pulling 
from your router configuration database and publish to your DNS 
database. If you are still doing either by hand then it’s time to make 
the switch ASAP.


Sent from my iCar


What's the reverse of your iCar? ;)

Re: looking for hostname router identifier validation

[Bryan Holloway]

On 4/29/19 7:21 PM, Valdis Klētnieks wrote:

On Mon, 29 Apr 2019 16:16:06 -0500, Bryan Holloway said:


I still see references to UUNet in some reverse PTRs.

So, uh, yeah.


I wonder what year we'll get to a point where less than half of NANOG's
membership was around when UUNet was. We're probably there already.
And likely coming up on when less than half the people know what it
was, other than myth and legend



Bought my first T-1 from those guys ... don't even ask how much it cost.

Re: NTP for ASBRs?

[Bryan Holloway]

On 5/8/19 4:00 PM, Scott Weeks wrote:



--- j...@ntt.net wrote:
From: Job Snijders 

on this topic, i strongly recommend to operate all
devices in the Etc/UTC timezone, this makes
coordination with external entities much easier.



Yes, this!  Holy crap I come upon a lot of networks
that don't do this and it's always painful.

scott



Now if only we could get rid of Daylight Saving Time ...

Re: NTP for ASBRs?

[Bryan Holloway]

On 5/8/19 6:54 PM, Scott Weeks wrote:



--- br...@shout.net wrote:
From: Bryan Holloway 
On 5/8/19 4:00 PM, Scott Weeks wrote:

--- j...@ntt.net wrote:
From: Job Snijders 

on this topic, i strongly recommend to operate all
devices in the Etc/UTC timezone, this makes
coordination with external entities much easier.



Yes, this!  Holy crap I come upon a lot of networks
that don't do this and it's always painful.


Now if only we could get rid of Daylight Saving Time ...
--

Luckily, Hawaii doesn't have that problem...

https://en.wikipedia.org/wiki/Daylight_saving_time_in_the_United_States#Hawaii

But, that's the thing.  One time.  No trying to figure
out who does DST and who doesn't.



100% true. But there is also a practical side to this ...

When a NOC-ling, in their own local timezone, says, "hey, what happened 
two hours ago?", they have to make a calculation. And that calculation 
annoyingly depends on the time of year in many if not most locales 
worldwide. And to make matters worse, some folks change at different 
times of the year, so, if you're a global network 


Hawai'i and Arizona can add/subtract without looking at the damn 
calendar. I'm just sayin' I'd like to see more of that.

Re: NTP for ASBRs?

[Bryan Holloway]

On 5/8/19 7:55 PM, Brian Kantor wrote:

On Wed, May 08, 2019 at 07:47:56PM -0500, Bryan Holloway wrote:

100% true. But there is also a practical side to this ...

When a NOC-ling, in their own local timezone, says, "hey, what happened
two hours ago?", they have to make a calculation. And that calculation
annoyingly depends on the time of year in many if not most locales
worldwide. And to make matters worse, some folks change at different
times of the year, so, if you're a global network 

Hawai'i and Arizona can add/subtract without looking at the damn
calendar. I'm just sayin' I'd like to see more of that.


Clocks are cheap. I have two on the wall; one is local time and
the other is marked GMT.
- Brian


Cheap != free. Many clocks have to be set after a DST change. Clocks 
that do this automatically are > cheap.


I stand by my point.

Disclaimer: I have two clocks.

Re: NTP for ASBRs?

[Bryan Holloway]

On 5/8/19 10:15 PM, Bryan Holloway wrote:



On 5/8/19 7:55 PM, Brian Kantor wrote:

On Wed, May 08, 2019 at 07:47:56PM -0500, Bryan Holloway wrote:

100% true. But there is also a practical side to this ...

When a NOC-ling, in their own local timezone, says, "hey, what happened
two hours ago?", they have to make a calculation. And that calculation
annoyingly depends on the time of year in many if not most locales
worldwide. And to make matters worse, some folks change at different
times of the year, so, if you're a global network 

Hawai'i and Arizona can add/subtract without looking at the damn
calendar. I'm just sayin' I'd like to see more of that.


Clocks are cheap. I have two on the wall; one is local time and
the other is marked GMT.
- Brian


Cheap != free. Many clocks have to be set after a DST change. Clocks 
that do this automatically are > cheap.


I stand by my point.

Disclaimer: I have two clocks.


And furthermore, GMT != UTC.

Re: Spamming of NANOG list members

[Bryan Holloway]

Anybody else noticed a significant uptick in these e-mails?

When I first saw this thread, I hadn't seen any. A couple days later, I 
got my first one. (yay!) Now I'm getting 2-3 a day. (yay?)

Re: DOs and DONTs for small ISP

[Bryan Holloway]

On 6/4/19 9:20 AM, Mark Tinka wrote:



On 3/Jun/19 15:41, Fletcher Kittredge wrote:


Here is your checklist in descending order of importance:

 1. market opportunity
 2. finding the right partners (see below)
 3. financial
 4. sales and marketing
 5. organizational capacity and HR
 6. legal, regulatory
 7. capital acquisition
 8. security
 9. ...
10. ...
11. ...
12. technical including equipment selection, routing policy,
filtering, etc



13. Don't run Mikrotik.

I'm kidding... I think :-)...

Mark.


14. Go with K56flex, not X2.

Re: CenturyLink/Level3 feedback

[Bryan Holloway]

On 6/5/19 3:40 PM, Dovid Bender wrote:
If the FCC has their way the only place you will see the PSTN in history 
books. I can only hope that the same happens to faxing.




I'm told that the one of the only reasons faxing is still a thing is 
because of HIPAA-compliance.

Re: few big monolithic PEs vs many small PEs

[Bryan Holloway]

On 6/21/19 10:01 AM, Aaron Gould wrote:

I was reading this and thought, planet earth is a single point of failure.

...but, I guess we build and design and connect as much redundancy (logic, hw, 
sw, power) as the customer requires and pays for and that we can truly 
accomplish.

-Aaron




I don't know about you, but we keep two earths in active/standby. Sure, 
the power requirements are through the roof, but hey -- it's worth it.

Re: Intermittent "bad gateway"

[Bryan Holloway]

Maybe Verizon can blog about it.


On 7/2/19 9:35 AM, Ryan Hagman wrote:

https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

One hell of a fall.

On Tue, Jul 2, 2019 at 10:33 AM Michael Rathbun > wrote:


On Tue, 2 Jul 2019 07:16:36 -0700, Stephen Satchell
mailto:l...@satchell.net>>
wrote:

 >Are we having another BGP problem this morning?

Cloudflare did fall over for a bit this morning.

mdr
-- 
    Sometimes half-ass is exactly the right amount of ass.

        -- Wonderella

Re: Phoenix IX down/gone?

[Bryan Holloway]

On 8/2/19 3:29 AM, Brandon Wade via NANOG wrote:

Corrected URL:
https://peeringdb.com/ix/662

Anyone know what happened to Phoenix IX? https://peeringdb.com/ix/66  They
seem off the air including website and phones.. permanently?

-PeterK at 32354  


We (AS53767) are passing traffic on Phoenix IX. The person who operates 
the exchange (Paul Emmons) is frequently traveling for business. I 
suspect something is simply wrong with the website.


I'll reach out to him through a private channel and make sure he's aware 
the website is down.


Best regards,
Brandon Wade
iCastCenter.com / AS53767



The web-site appears to be up again.

Re: MAP-E

[Bryan Holloway]

On 8/2/19 5:16 PM, Baldur Norddahl wrote:


Multiple customers share an IPv4 address each with an assigned port range.




One downside that has been brought up on the list before is that a DDoS 
attack against a single subscriber will impact many, but that particular 
drawback may not outweigh the benefits (costs).

Re: rr.level3.net on autopilot?

[Bryan Holloway]

AS1, baby!!


On 9/6/19 1:45 AM, Bryan Fields wrote:

On 9/5/19 2:05 PM, Jon Lewis wrote:

I was doing some IRR clean-up and after a few successful updates, I'm no
longer able to alter or delete our objects in rr.level3.com.

Emails to r...@level3.com result in no action and no response.  I've tried
reaching out to the Level3 (Centurylink) NOC via email and phone, and
can't seem to find anyone who knows what rr.level3.com is, much less knows
who to talk to about troubleshooting.  Anyone know who (if anyone) keeps
the wheels spinning on the Level3 IRR?


The other day I tried to clean up some old entries from the 2000's and Genuity
entries that became part of it.  This was a failure, the NOC knew nothing
about it, and worse didn't get my black rocket jokes. No one working there
knew what Genuity was.

I gave up.

Re: sfps from fs dot com

[Bryan Holloway]

On 9/20/19 2:31 PM, Nicholas Warren wrote:

Anyone have experience with fs.com's lasers? Are they reliable?



YMMV.

Re: sfps from fs dot com

[Bryan Holloway]

It boils down to a business case.

In my travels we see a high failure rate -- higher than I'd like to see 
--, but $boss likes the price, and, as Jason pointed out below, for the 
price, it can be a "successful" business model.


In a nutshell:

For someone on a budget, they're great; buy spares.

For someone who doesn't want to deal with outages, call-volume, hands 
tickets, truck-rolls, and has money to burn, there are better alternatives.



On 9/20/19 2:41 PM, Jason Lixfeld wrote:

We have maybe 1:50 DOA, but they’re so cheap, we just throw them out because 
it’s not worth the RMA.

That said, I can’t remember the last time we’ve had any of these fail in the 
field, or have had any issues with variablity in TX/RX power.  We have tens of 
thousands of these in the field, from 100Mb Bidi to 100G-LR and everything in 
between, including *WDM up to 80KM.


On Sep 20, 2019, at 8:31 AM, Nicholas Warren  wrote:

Anyone have experience with fs.com's lasers? Are they reliable?

Twitter contact?

[Bryan Holloway]

Anyone from Twitter lurking? Trying to resolve a peering issue and not 
getting far through published contacts.


Thanks!
- bryan

Re: Twitter contact?

[Bryan Holloway]

Someone has reached out; I’m good!

> On Oct 21, 2019, at 13:54, Bryan Holloway  wrote:
> 
> Anyone from Twitter lurking? Trying to resolve a peering issue and not 
> getting far through published contacts.
> 
> Thanks!
>- bryan

Re: Disney+ Streaming

[Bryan Holloway]

On 11/13/19 1:06 PM, Niels Bakker wrote:

* mikeboli...@gmail.com (Mike Bolitho) [Wed 13 Nov 2019, 12:05 CET]:

This has gone well beyond out of scope of the NANOG list. Discussing who
watches what kind of content has nothing to do with networking. Can you
guys take the conversation elsewhere?


On the contrary.  This discussion informs eyeball networks' capacity 
planning requirements for the upcoming years.


It'd be nice to go from anecdata to data, though.


 -- Niels.



Indeed ... as an eyeball network, this is all very relevant.

Another aspect that hasn't been mentioned in this thread (I think), is 
that besides there being a potential saturation of streaming services, 
there's also the backroom dealings between content and content-providers.


Here's some data: Netflix just lost "Friends", one of its most popular 
offerings (and probably more than a blip on my bandwidth graphs) to HBO 
Max. This is but one example, but, as a whole, stuff like this is very 
important for capacity-planning.


Not saying it's gonna happen, but if Disney "lost" the Star Wars 
franchise to, say, Amazon, you better believe there are likely to be 
traffic shifts. (Yes, I know they own it.)

AS12042 contact?

[Bryan Holloway]

If there's anyone from Consolidated (legacy Enventis) lurking, could you 
please reach out to me off-list?


You're advertising a prefix from a Tier 1 to another Tier 1 which is 
causing occasional issues for us.


E-mails to noc@ and repair@ have gone unanswered (and admittedly I'm not 
a direct customer.)


Thank you ...

Re: AS12042 contact?

[Bryan Holloway]

On 11/25/19 8:50 PM, Bryan Holloway wrote:
If there's anyone from Consolidated (legacy Enventis) lurking, could you 
please reach out to me off-list?


You're advertising a prefix from a Tier 1 to another Tier 1 which is 
causing occasional issues for us.


E-mails to noc@ and repair@ have gone unanswered (and admittedly I'm not 
a direct customer.)


Thank you ...



This has been resolved -- thanks to all who reached out!

Re: Elephant in the room - Akamai

[Bryan Holloway]

On 12/5/19 8:48 AM, Matthew Petach wrote:



On Wed, Dec 4, 2019, 19:05 Kaiser, Erich > wrote:


Lets talk Akamai


[...]


The last two nights the traffic levels to them has skyrocketed as well.

Any insight?


Erich Kaiser
The Fusion Network


As a CDN, I would usually expect to see traffic *from* Akamai to be the 
large direction.


If you're seeing your traffic *to* them skyrocketing, are you sure you 
aren't carrying DDoS attack traffic at them?


CDNs aren't known for being large traffic sinks.   ^_^;;

Matt



I think he meant inbound (from). We also saw the same thing.

Re: Software Defined Networks

[Bryan Holloway]

On 12/5/19 6:16 PM, Patrick W. Gilmore wrote:

I tell everyone we had SDNs in the 90s.

But we called it “expect scripts”.

:-)

--
TTFN,
patrick



I miss TCL ...

10G tester recommendations?

[Bryan Holloway]

We're in the market for a hand-held 10G ethernet tester, and I was 
curious if the NANOG community had any recommendations or experiences 
they would be willing to share, negative or positive.


Currently we're looking at the EXFO MAX-800 and NetScout's AT 10G, but 
we're open to other suggestions.


Thank you!

- bryan

Re: 10G tester recommendations?

[Bryan Holloway]

Mainly RFC2544 and physical media. QoS could be handy, but it's not as 
important as proving that the circuit is meeting our expectations from 
the carrier(s).



On 10/4/16 1:12 PM, Saku Ytti wrote:

On 3 October 2016 at 22:09, Bryan Holloway  wrote:

We're in the market for a hand-held 10G ethernet tester, and I was curious
if the NANOG community had any recommendations or experiences they would be
willing to share, negative or positive.


What are you looking to test? Are you interested in having the kit
emulate many devices? Do you want to be able to test QoS
realistically? Do you care about one-way latency accurately?
Or is this more about proving the physical media is fine?

Re: Help interpret a strange traceroute?

[Bryan Holloway]

On 10/31/16 4:20 PM, Olivier Benghozi wrote:

Hi Randy,


ECMP loadbalancing is most frequently done on layer3+layer4 headers, and 
unixlike traceroute use UDP with increasing destination port number for each 
packet (usually starting at 33434), which allows to see the different available 
paths, as wrote William.

Would you want/need to stick to only one traceroute path, you may use ICMP 
traceroute instead of UDP traceroute (no port in ICMP, so only layer 3 
available to loadbalance, so all packets will go through the same interface).

Usually it is achieved by using traceroute -I yourdest
Windows tracert is ICMP only traceroute by the way. MTR tool is also ICMP based 
by default.

Keep in mind that it looses some useful information, though (since you see only 
one path and don't decide which).
So, you can also use UDP traceroute with fixed port (by example 33434 with no 
port increase), and try again the same traceroute with another destport (with 
fixed port too, by example 33435), which would display two different paths in a 
more readable way. RTFM is required since the options depend on your traceroute 
particular specie :)


Olivier


On 31 oct. 2016 à 20:42, William Herrin  wrote :

On Mon, Oct 31, 2016 at 3:33 PM, Randy  wrote:

Any idea how a traceroute (into my network) could end up this fubar'd?
Discovered this wierd routing while investigating horrendously slow speeds
(albeit no packet loss) to a particular ISP abroad.


Hi Randy,

This is per-packet load balancing. In the forward path the alternates
are different lengths but the traceroute stops as soon as at least one
of the paths reaches the destination.

The return path is also engaged in per-packet load balancing but the
paths are all the same length.




This is also a handy tool that addresses the same issues:

https://paris-traceroute.net/

Re: Comcast business IPv6 vs rbldnsd & PSBL

[Bryan Holloway]

I concur with the kudos bit, but I'll also concur that the CPE support 
appears to be limited. Another example: IPv6 prefix delegation is broken 
on the SMCD3G-CCR, and according to the following threads:


http://www.gossamer-threads.com/lists/nsp/ipv6/54761 (scroll down to the 
IPv6 OPERATIONS - BUSINESS section)


http://forums.businesshelp.comcast.com/t5/IPV6/Dual-Stack-on-SMC-D3GCCR-and-Cisco-DPC3939B/td-p/20504

... others have the same issue and there isn't much of an incentive to 
fix it.


When I asked if I could use my own CPE, I was told no, because I'm a 
"business customer", which is a requirement if you want static v4 IPs.


Anyone have any success with a different model CPE and Comcast v6? I 
love that they hand out a /56 by default, but it's not of much use if I 
can only use a single /64.


- bryan


On 11/29/16 11:45 AM, Livingood, Jason wrote:

I can send it along to folks here at Comcast.

- Jason

On 11/28/16, 1:46 PM, "NANOG on behalf of Rik van Riel"  wrote:

First of all, kudos to Comcast for trying to roll out IPv6 across
their entire network. Static IPv6 netblocks seem to be available
for Comcast business users, and IPv6 is enabled unconditionally
in the CPE routers used by Comcast business class internet.

Unfortunately, the software in the two available CPE routers
(SMC & Cisco) is horribly broken when it comes to IPv6.

The TL;DR summary: even when IPv6 firewalling is disabled in
the configuration, the router still tracks every IPv6 "connection",
which causes every single DNS lookup to fill up a slot in its
connection tracking table.

The router's logs say it blocks tens of thousands of IPv6
connections every day, despite firewalling being "disabled" on
the router.

Once the connection tracking table fills up, both IPv6 and IPv4
start having trouble, with packet loss on ICMP, high ping times
to the local router (and the internet), and new connections not
establishing. The router randomly crashes and reboots too,
sometimes multiple times a day.

This ends up breaking both IPv6 and IPv4.

It only takes about 300kbit/s of DNS traffic to trigger the bug,
in both the SMC and the Cisco routers.

Are there any Comcast NOC or other technical people present who
could help?

I am interested both in helping resolve the firmware issues in
the routers (there will no doubt be other customers who hit this
in the future, as IPv6 becomes ore common) or, if that is not an
option, finding some way to avoid the issue.


http://forums.businesshelp.comcast.com/t5/Equipment-Modems-Gateways/Cis
co-DPC3941B-slows-to-a-crawl-and-crashes-several-times-a-day/td-p/30807

--
All Rights Reversed.

Re: Comcast business IPv6 vs rbldnsd & PSBL

[Bryan Holloway]

Not to mention that they "raised my rent" a few months ago by $5/mo, 
which is pretty ludicrous considering that a) it doesn't actually work 
as advertised, and b) it probably cost them $20-30 to purchase those 
SMCs wholesale in the first place. They've made their money on my CPE 
many many times over.


But that's just the way it is.


On 11/29/16 1:48 PM, Luke Guillory wrote:

Because if you want static IPs from them you must rent one of the following.

Cisco DPC3939B or DPC3941B
Netgear CG3000DCR
SMC Networks SMCD3G




Luke Guillory
Network Operations Manager

Tel:985.536.1212
Fax:985.536.0300
Email:  lguill...@reservetele.com

Reserve Telecommunications
100 RTC Dr
Reserve, LA 70084

_

Disclaimer:
The information transmitted, including attachments, is intended only for the 
person(s) or entity to which it is addressed and may contain confidential 
and/or privileged material which should not disseminate, distribute or be 
copied. Please notify Luke Guillory immediately by e-mail if you have received 
this e-mail by mistake and delete this e-mail from your system. E-mail 
transmission cannot be guaranteed to be secure or error-free as information 
could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or 
contain viruses. Luke Guillory therefore does not accept liability for any 
errors or omissions in the contents of this message, which arise as a result of 
e-mail transmission. .

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Bush
Sent: Tuesday, November 29, 2016 1:41 PM
To: Rik van Riel
Cc: North American Network Operators' Group
Subject: Re: Comcast business IPv6 vs rbldnsd & PSBL

i am running my own (why rent at silly costs) dpc3008 and wfm.

randy