Re: DSLAM

[Eric]

I can do it. 

Sent from my iPhone

> On Mar 3, 2014, at 12:40 PM, "Nick Olsen"  wrote:
> 
> Hey Guys, I need a 24 port ADSL (2, +, It's all the same in my book) DSLAM. 
> And I need it by tomorrow.
> 
> Normal channels seem to be impacted by weather. Not to mention we've been 
> pretty unhappy with our current models (Versa, And Planet).
> 
> Any options? Unicast is acceptable.
> 
> Nick Olsen
> Network Operations  (855) FLSPEED  x106
> 
> 

Cogent - ATT issue?

[Eric]

Anyone know if there is a connectivity issue between Cogent and ATT in the 
northeast?  We're seeing random timeouts to some systems we have in an ATT data 
center but only from sources on Cogent's network.

Thanks... 

- Eric :)

Re: IP Address Management IPAM software for small ISP

[Eric]

I ran Zenoss for a network with about 5k - 7k switches/APs, about 100 L3 
devices (routers, firewalls), and about 50 servers/appliances without any 
polling problems.  This was a few years ago on the open source product.  With 
that said, we were reluctant to expand this to monitor the rest of our 
enterprise as the open source version didn't support distributed 
polling/collection, which might be the scaling issue Jo mentioned...  That, 
unfortunately, was only available in the paid "enterprise" one.  Other than 
that, we really liked it.



On Dec 25, 2012, at 1:57 AM, Mike Hale  wrote:

> Ahhh.
> 
> That sucks.  I've never put our Zenoss installs through quite that much
> traffic.  That's a shame to hear.
> 
> 
> On Mon, Dec 24, 2012 at 10:53 PM, Jo Rhett  wrote:
> 
>> Small shop people wise with millions of customers and tens of thousands of
>> application and log-derived data sources. We use Zenoss extensively and
>> mostly we keep having to make decisions what data to pull out of it so it
>> can function.
>> 
>> I have previously worked at larger enterprises which had millions of data
>> sources, and Zenoss couldn't dream of handling that, no matter how much
>> hardware we threw at it.
>> 
>> 
>> On Dec 24, 2012, at 10:48 PM, Mike Hale wrote:
>> 
>> Very small shop with millions of data sources?
>> 
>> lol?
>> 
>> 
>> On Mon, Dec 24, 2012 at 10:38 PM, Jo Rhett wrote:
>> 
>>> On Dec 20, 2012, at 9:26 PM, Charles N Wyble wrote:
 Zenoss works very well
>>> 
>>> Um... you lost me after the first 4 words. Zenoss might work acceptably
>>> for very, very small organizations with very small amounts of data. Zenoss
>>> is incapable of scaling to even moderate-sized data sets with tens of
>>> thousands of data sources, nevermind medium sized data sets with millions
>>> of data sources. I work at a very small shop with three total engineers and
>>> Zenoss was unable to scale beyond 1/4 of our data sources with dozens of
>>> cores and hundreds of gigabytes of RAM on numerous systems.  It doesn't
>>> actually use any of these, the internal deadlocks in the architecture make
>>> it impossible for it to scale.
>>> 
>>> That Zenoss might make a better IP management tool than what it is
>>> purported and sold to do... amuses.
>>> 
>>> --
>>> Jo Rhett
>>> Net Consonance : net philanthropy to improve open source and internet
>>> projects.
>> 
>> 
>> --
>> 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
>> 
>> 
>> --
>> Jo Rhett
>> Net Consonance : net philanthropy to improve open source and internet
>> projects.
> 
> 
> -- 
> 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

Berlin ISP

[Eric]

Hello,

I'm looking for about a 10-20mbps ISP circuit for our Berlin office.  Any 
recommendations on who provides access there and might be able to deal with us 
in English?

Eric

Re: Is AS information useful for security?

[Eric]

It's useful in terms of remediation as it can help identify through which 
"door" packets entered your  network.  Though, as others will undoubtedly point 
out, it's trustworthiness will depend upon how you derive the AS mapping and 
upon other security features  (e.g. uRPF)

-- Eric :)


> On Thu, 15 Dec 2011, Joe Loiacono wrote:
> 
>> Is a good knowledge of either origin-AS, or next-AS with respect to flows
>> valuable in establishing, monitoring, or re-enforcing a security posture?
>> In what way?

Re: IP Management Software

[Eric]

you didn't specify "open source"' so I'll throw out IPControl by BT/INS.  I 
used it at my last place to manage about 100k+ DNS entries (3x /16s, misc 
blocks, RFC1918) and our DNS/DHCP servers.  Worked great but not cheap :)

-- Eric :)

On Dec 16, 2011, at 4:46 PM, deles...@gmail.com wrote:

> Not to be a bandwagon jumper but +1 for 6connect as well.
> --Original Message--
> From: Mike Walter
> To: nanog@nanog.org
> Subject: RE: IP Management Software
> Sent: Dec 16, 2011 4:42 PM
> 
> +1, agree on 6connect.net.  
> 
> -Original Message-
> From: Rafael Rodriguez [mailto:packetjoc...@gmail.com] 
> Sent: Friday, December 16, 2011 12:55 PM
> To: Shahab Vahabzadeh
> Cc: nanog@nanog.org
> Subject: Re: IP Management Software
> 
> Check out 6connect.
> 
> Sent from my iPhone
> 
> On Dec 16, 2011, at 11:03, Shahab Vahabzadeh  wrote:
> 
>> Hi everybody,
>> Can anybody share his/her experience with IP Management software's? Which I
>> can use it managing near 100K IP Address?
>> IPPlan is not good enough, I think its covering all my need and not fully
>> flexible.
>> If you have discuss this before here please share me the link.
>> Thanks
>> 
>> -- 
>> Regards,
>> Shahab Vahabzadeh, IP Engineer, *nix Admin and Geek
> 
> 
> 
> 
> Sent from my BlackBerry device on the Rogers Wireless Network

Re: did AS174 and AS4134 de-peer?

[Eric]

+1

- Eric

On Mar 7, 2012, at 7:37 PM, Michael Sinatra  wrote:

> On 03/07/12 16:10, Patrick W. Gilmore wrote:
>> On Mar 7, 2012, at 19:06 , Jim Cowie wrote:
>> 
>>> As a meta-comment: this "Quick Look" style of blog is an experiment we're 
>>> trying, based on feedback that the community wanted to hear about more of 
>>> these little events as they happen.  In a Quick Look, we're giving the 
>>> facts as they are known from initial measurement, and a very quick summary 
>>> of our preliminary analysis of the incident.   Then we throw the topic open 
>>> to comments from those who might have the clues to the rest of the story ...
>> 
>> Well, this member of the community appreciates it.
>> 
> 
> +1
> 
> I find the combination of facts and inferences presented to be interesting 
> and useful.
> 
> michael

Securing OOB

[Eric]

Hello,

It seems that the current practice is to use a DSL line, as opposed to a modem, 
for accessing an OOB a console server at a remote colo.  From a security 
standpoint, what do people generally do - trust the console server, repurpose 
an old linksys box from my house or put in a full firewall?  

Eric :)

Android lack of DHCPv6 purchasing decisions?

[Eric]

Hi All,

I have a question for people that deal with mobile devices in
enterprise. Have you decided not to purchase Android devices due to the
lack of DHCPv6 support and consider Apple or some other vendor devices
instead?

It's been thrown around here, discussed and it's absurd so I'm curious
what business purchasing decisions it has lead to now.

Thanks

Re: Network diagnostics for the end user

[Eric]

+1. It's especially helpful for wireless troubleshooting in a campus 
environment.  You can get a lot of info from the AP, but tend not to know what 
the client is seeing and it's great for catching transient events (oh, whenever 
the elevator goes by...)

Eric


On Jun 22, 2013, at 12:29 AM, "Carlos M. Martinez"  
wrote:

> May sound silly, but in another life I faced a similar problem and by
> hosting local SpeedTest.net servers in our network we could fend off
> many of these calls.
> 
> But I guess it will depend on your customers, whether they take it or not.
> 
> cheers,
> 
> ~Carlos
> 
> On 6/20/13 9:45 PM, Jeffrey Ollie wrote:
>> Are there any tools out there that we could give to our end users to help
>> diagnose network problems? We get a lot of "the Internet is slow" support
>> calls and it would be helpful if we had something that would run on the end
>> user's computer and help characterize the problem. We have central
>> monitoring system of course but that doesn't always give a complete
>> picture, as the problem could always be on the end user's computer - slow
>> hard drive, not enough memory, wrong name servers, etc.
> 

Re: Exchange Point

[Eric]

There's also the bug R&E data center being built in Holyoke and I'm guessing 
most fiber runs back to people who put stuff there will go via Springfield...  
Might be of interest to people who don't have/want dark fiber all the way...  I 
also think (but might be off) that OSEAN and some other regional state & R&E 
networks have a presence there.



On Aug 10, 2013, at 7:21 PM, Bill Woodcock  wrote:

> 
>>> Does anyone see value in putting a open exchange point in one federal in
>>> Springfield, MA?
>> 
>> Springfield, Mass.?  Can't hurt…
> 
> Agreed.  There's essentially no reason not to put in an exchange, anywhere 
> three or more ISPs are present.
> 
>> There's also Boston Internet Exchange which has been slowly picking up
>> steam, over at 1 Summer; you may want to look into remote interconnection.
> 
> That's 90 miles away, so I'd recommend not trying to extend their switch 
> fabric that far.  If you were considering a second exchange in Boston, I'd 
> definitely recommend using the same switch fabric for both locations, but not 
> at 90 miles of remove.  That would put a pretty big burden on the smaller 
> ISPs that are likely to be participating in Springfield.
> 
>-Bill
> 
> 
> 
> 

Re: CenturyLink/Level 3 combined AS

[Eric Flanery (eric)]

On Fri, Jun 7, 2019 at 10:03 AM Romeo Czumbil 
wrote:

> All new CL Internet get's provisioned on AS3356
> You would need a strong case for them to put you on AS209
>
> At this time they are not merging the two AS's
> And also define "quality" ;-)
>
> -Romeo
>

That isn't true in my recent experience. We just replaced an older L3
transit with a new CL one on different floors of the same building, and
doing so involved moving from 3356 to 209.

Interestingly, the CFA for the new circuit's attachment suggests it came
out of the "L3" suite, not the "Qwest" suite (both on the same floor); so
it seems that 209 is provision-able at former L3 facilities.

Other recent entirely new CL turn-ups with us, out of rural COs belonging
to Frontier, have also been with 209.

--Eric

Re: ISP License in the USA?

[Eric Flanery (eric)]

There is no such thing as an 'ISP license' in the US. I have a hard time
imagining Texas of all places would have such a requirement.

Depending on what exactly you are doing, there are various and highly
varied requirements, such as acquiring a SPIN number for E-Rate, filing FCC
477 if you do broadband, FCC 499 if you do VoIP (CLEC and ETC also apply
there), a FRN if you do pretty much anything FCC-related, various sorts of
licenses for most radio/microwave systems (excepting part 15 stuff), CALEA,
open internet, etc...

COALS _could_ apply _if_ you are running a cable TV system that also
delivers data services, but it isn't an 'ISP thing'.

More to the point...

I wouldn't take US legal advice from any consultant not familiar with US
law, or really any non-lawyer consultant at all. I wouldn't take it from
NANOG either; while it's a tremendous technical resource, it is not your
attorney.

There are a number of telecommunications focused law firms out there, with
knowledgeable lawyers. It would be a good idea to establish a relationship
with one, if you intend to enter the increasingly complex legal minefield
of being an ISP.

--Eric

On Tue, May 31, 2016 at 11:24 AM, Dan White  wrote:

> Not familiar with the process, but look at E-rate if you want to provide
> service to schools, libraries and health providers.
>
>
> On 05/31/16 13:14 -0500, Lorell Hathcock wrote:
>
>> NANOG:
>>
>> Our owner has hired a consultant who insists that we should have an ISP
>> license to operate in the United States.  (Like they have in other
>> countries
>> like Germany and in Africa where he has extensive personal experience.)
>>
>> I am asking him to tell me which license we should have because I don't
>> know
>> of a license that we are required to have to route IP traffic to end
>> customers.
>>
>> I am familiar with CLEC status filed with our state.  But it is not a
>> requirement to pass traffic.
>>
>> He is suggesting COALS with which I am completely unfamiliar.
>>
>> Can anyone tell me if there is a Texas state and/or USA Federal license
>> for
>> a small operator to pass IP traffic from the internet to end users
>> (commercial and/or residential).
>>
>> I am aware that there are some CALEA requirements of ISPs that seem to
>> kick
>> in once a CALEA request is made, but is that different from a license.
>>
>
> --
> Dan White
> BTC Broadband
>

Re: ISP License in the USA?

[Eric Flanery (eric)]

These are the two I'm most familiar with:

Lerman Senter, as Faisal mentioned: http://www.lermansenter.com/

Rini O'Neil: http://rinioneil.com/

--Eric

Re: carrier comparison

[Eric Flanery (eric)]

Vlade,

When you say that "they still advertise your routes", do you mean:

A: That you were having them originate your routes, and they failed to stop
doing so when they had problems? Or...

B: That routes you were originating continued to be propagated by them,
even though your session with them was down? Or...

C: Something else.

I ask, as we are considering some cheap Cogent bandwidth in the
not-too-distant future, to allow us to keep commit rates low on higher
quality connections. 'A' wouldn't be a real problem, since we run our own
AS and originate our own routes; 'B' could be potentially devastating.


On Thu, Feb 6, 2014 at 8:04 AM, Vlade Ristevski  wrote:

> We have had Cogent over Verizon's Fiber for more than a few years now.
> Cogent goes down once at year at minimum. They had 2 outages in a single
> day a couple days ago in Northern NJ.  One in the AM "..caused by a power
> outage in a vendor data center where Cogent is collocated." They went on to
> have another outage at around 9:30 PM on the same day for which I'm still
> waiting for an RFO. During this outage, they still were advertising our BGP
> routes so we didn't fail over to our 2nd provider. I notice that happens
> alot with them. When they go down, they still advertise your routes.
>
> As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
>
> Our college is kind of far from things so we don't have a lot of outside
> fiber coming. The last mile fiber for both of our connections are different
> from our Internet providers. I've never had a big issue with the two
> working with each other. The only issue we had is I suspected we weren't
> getting as much bandwidth as we paid for. They had to work out where the
> policer and/or bottle neck was. This is the only issue we had in 5 years
> with this set up and it got resolved. IME, when there is a full outage,
> it's always been clear who the responsible party is.
>
>
>
>
>
>
> On 2/6/2014 10:17 AM, Adam Greene wrote:
>
>> Hi,
>>
>>
>> We're a small ISP / datacenter with a Time Warner fiber-based DIA contract
>> that is coming up for renewal.
>>
>>
>> We're getting much better pricing offers from Cogent, and are finding out
>> what Level 3 can do for us as well. Both providers will use Time Warner
>> fiber for last mile.
>>
>>
>> My questions are:
>>
>> -  Will we be sacrificing quality if we spring for Cogent?
>> (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
>>
>> -  Is there a risk with contracting a carrier that utilizes
>> another
>> carrier (such as Time Warner) for the last mile? (i.e. if there is a
>> downtime situation, are we going to be caught in a web of confusion and
>> finger-pointing that delays problem resolution)?
>>
>> -  How are peoples' experiences with L3 vs TWC?
>>
>>
>> Although I assume everyone on the list would be interested in what others
>> have to say about these questions, out of respect for the carriers in
>> question, I encourage you to email frank opinions off list.
>>
>>
>> Or if there are third party tools or resources you know that I could
>> consult
>> to deduce the answers to these questions myself, they are most welcome.
>>
>>
>> Thanks,
>>
>> Adam
>>
>>
>

Re: MACsec SFP

[Eric Flanery (eric)]

Hmm, wandering pie-in-the-sky module wish list...

MACsec would be great, hopefully in an easy to manage/replace form.

Separately tunable transmitters and receivers; in both DWDM and CWDM
flavors. This would reduce the number of different parts to track/stock,
and enable the use of simple splitters/combiners in place of mux/demux
shelves for some short links.

Fully functional (as a manageable device) whatever-PON 'OLT in a module',
so that we can use any old host device that lacks specific support. (ONTs
too)

'Channelized SFP+', that talks on multiple wavelengths (CWDM/DWDM) at a 1G
rate simultaneously, to support a sort of WDM-PON. And/or SFP+ modules that
talk on multiple strands at a 1G rate simultaneously, with something like a
MPO/MTP connector, to increase density. I suppose there would need to be
some sort of switch or mux built into the module in either case.

A SFP(+) microwave modem, to connect to a radio head. Hopefully with wide
support of many licensed and unlicensed bands.

802.11-whatever APs in a SFP(+) form factor, preferably controller-based.
Perhaps doing some sort of RF-over-Ethernet to enable widely distributed
antenna systems.

Mini MPLS LER/PE routers in SFP form. All sorts of customer-facing
interface types could be interesting, but mostly (sub-rate) Ethernet with
QoS of some sort.

Self power-leveling and/or attenuating with wide ranges, again reducing
part tracking/stocking, and eliminating discrete attenuator pads.

SIP ATA in SFP form.

Small flash-based NAS in SFP form.

'Computational resources' in SFP(+) form (ARM chips maybe?).

OTDR functionality built into modules, hopefully able to operate without
disrupting the data flow, perhaps continuously.

Manageable (CLI and SNMP, please) modules of all sorts, to enable whatever
'special' features to be operated without requiring any particular support
from the host device beyond the MSA.

1G/100M SFPs that provide PoE ('passive' 18v or 24v would be most
appreciated.)

No vendor lock!

--Eric


On Tue, Jun 24, 2014 at 10:19 AM, Saku Ytti  wrote:

> On (2014-06-24 12:30 -0400), Christopher Morrow wrote:
>
> > it's going to be hard to schedule a key roll then, right? I would
> > expect that in most/many deployments where someone enters a 'key'
> > there has to be some compliance process that includes: "And you change
> > that key every X days" right? So you'll NOT want to be in a situation
> > that involves coordinating a few thousand truck rolls every X months
> > to have this deployed.
>
> Hopefully you could offer date when new keys take effect.
>
> > > Maybe some customer would then enter need for this in CLI in their
> multimillion
> > > dollar RFQ, and then we'd get the feature.
> >
> > maybe so... multi-million of sfp is a lot of sfp though.
>
> Of course this would be for the equipment where SFP sits, SFP vendor can't
> solve this. But if you're making it mandatory in router RFQ, it seems
> pretty
> much guaranteed vendors would comply and winning bid at least would
> implement
> it.
>
>
> --
>   ++ytti
>

Re: MACsec SFP

[Eric Flanery (eric)]

Those 'proposals' are really just things that would have been useful in
module form at one point or another, not necessarily anything that I've
given any serious thought to what sort of market they would have. Some are
probably impractical, some would probably be far too expensive to actually
be useful, and some would probably only have very narrow appeal.

That said, I do think the separately tunable tunable transmitters and
receivers could be huge, especially if they came at only a reasonably small
premium over fixed wavelengths. Just for CWDM as we are implementing it, we
need to deal with 16 different wavelengths, and three different transmit
powers, giving us 48 different modules to deal with (DWDM would/will only
make that worse). If we could cut that to one, or even three, it would make
things much simpler, from planning to stocking and sparing.

--Eric


On Wed, Jun 25, 2014 at 1:55 AM, Pieter Hulshoff 
wrote:

> That's a large number of proposals. :) I'll have a chat with some
> colleagues about the types outside my areas of expertise to see what they
> think about it.
>
> You're not the first to mention separately tunable transmitters and
> receivers. Do you think there would be a great demand for this device?
>
> Anyone else care to wager in on these proposals; do any of them strike you
> as something you would be interested in as well?
>
> Kind regards,
>
> Pieter Hulshoff
>
>

Re: Optical Transport Platform

[Eric Flanery (eric)]

I've been really happy with the Fiberstore muxes/demuxes, although I'm
using CWDM not DWDM.

They do the job, are entirely passive (no power needed), perform well (real
world losses seem to match the test reports to a tee), seem solid enough,
and this sort of stuff doesn't get any cheaper (that I've found).

If you are looking for a 'platform', with _any_ sort of bells or whistles,
they aren't what you are looking for. but, if you just want a cheap way to
squeeze multiple channels onto a strand or two, they rock.

--Eric


On Mon, Aug 25, 2014 at 1:36 PM, Romeo Czumbil  wrote:

> Check out Ekinops
> They have been rock solid with me. They are not as expensive as the others
> and there's no switching in the backplane. Everything happens on the module
> cards
> http://www.ekinops.net
>
>
>
>
>
>
> Romeo Czumbil
> Sr. Network Engineer
>
>
> 1000 Adams Ave | Norristown, PA 19403
> tel 610-994-3046 | cell 609-220-0322
> rczum...@xand.com
>
> website  |  news  |  blog  |  data centers
>
> Confidentiality Note: This e-mail and any attachments are confidential and
> may be protected by legal privilege. If you are not the intended recipient,
> be aware that any disclosure, copying, distribution or use of this e-mail
> or any attachment is prohibited. If you have received this e-mail in error,
> please notify us immediately by returning it to the sender and delete this
> copy from your system. Thank you for your cooperation.
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Kenneth McRae
> Sent: Monday, August 25, 2014 3:59 PM
> To: NANOG
> Subject: Optical Transport Platform
>
> Greetings Everyone..
>
> Here is one for the transport/backbone/data center guys.  Can you tell
> which manufactures you like for optical transport?  I am looking for a
> solution to provide 8 channels of DWDM using tuned optics in my network
> gear. DWDM transport will be used for multiplexing only (no amplification
> or wavelength generation).  So far, I have been looking at Fiberstore and
> PacketLight..
>
> Any suggestions?
>
> All feedback is greatly appreciated!
>
> Thanks
>
> Kenneth
>

Re: Microwave link capacity

[Eric Flanery (eric)]

There is no simple answer, as the characteristics of each link are unique,
thus the requirements for each potential upgrade are also unique.

Typically an engineering study will be done to determine what exactly is
required, and what the cost will be. It could be as simple and cheap as a
software license upgrade costing a few hundred dollars; to a complete
tear-down and re-build of the towers (to support much larger antennas, for
example), costing hundreds of thousands. It can even involve adding
additional sites as relays, potentially pushing the cost into the millions.

--Eric


On Mon, Apr 4, 2016 at 10:28 AM, Jean-Francois Mezei <
jfmezei_na...@vaxination.ca> wrote:

>
> In a context of providing rural communities with modern broadband.
>
> Reading some tells me that Microwave links can be raised to 1gbps. How
> common is that ?
>
> I assume that cell phone towers have modern microwave links (when not
> directly on fibre). What sort of capacity would typically be provided ?
>
> And in the case of a remote village/town served by microwave originally
> designed to handle just phone calls, how difficult/expensive is it to
> upgrade to 1gbps or higher capacity ? Just a change of radio ? or radio
> and antenna, keeping only the tower ?
>
> (keeping spectrum acquisition out of discussion as that is a whole other
> ball game).
>

Re: ZyXEL Gear

[Eric Flanery (eric)]

I'll add that if you are comfortable with MikroTik, and can wait a few
months, they have announced a device with 12 SFP slots, and one SFP+ slot.

It's the CCR1016-12S-1S+, and I expect it to come in well under $1k.

--Eric (not OP)


On Wed, Nov 27, 2013 at 8:32 AM, Andrew D Kirch  wrote:

> Eric,
>
> I'll note as a followup to the Ipv6 thread, I'm a _HUGE_ mikrotik fan.
>  One of the CCR models has 4 SFP's.
>
> Andrew
>
>
> On 11/26/2013 10:47 PM, Eric C. Miller wrote:
>
>> I'm looking at some non-Cisco price options to deliver more than 4 SFP
>> slots into a structure and was wondering if anyone had any experience with
>> ZyXEL's offerings in the service provider market. Specifically MGS-3712F or
>> GS-4012F
>>
>> Thank you for your comments!
>>
>> Eric Miller, CCNP
>> Network Engineering Consultant
>> (407) 257-5115
>>
>>
>>
>>
>
>

Residential GPON last mile for network engineers (Telus AS852 and others)

[Eric Kuhnke]

With the growth of gigabit class single fiber GPON last mile services, I
imagine a number of people reading the list must have subscribed to such by
now.

Something that I have observed, and shared observations with a number of
colleagues, is that very often a person who works for ($someAS) lives in a
location where you are effectively singlehomed to ($someotherAS). Maybe you
bought your house before you got a job with your current employer, or maybe
the network you work for doesn't do residential last mile service at all.
Perhaps you work remotely for a regional sized entity that's a long
distance away from where you live.

Therefore necessitating a choice of service from whatever facilities based
consumer-facing ISP happens to service your home.

For example, in Seattle, a number of people discovered that they could keep
the Centurylink GPON ONT, and remove the centurylink-provided router/modem
combo device. Provided that they were able to configure their own router
(small vyatta, pfsense box, mikrotik, whatever) to speak a certain VLAN tag
on its WAN interface and be a normal PPPoE / DHCP client.

I'm sure there are a lot of people who prefer to run their own home router
and wifi devices, and not rely upon a ($big_residential_isp) provided
all-in-one router/nat/wifi box with opaque configuration parameters, or no
ability to change configuration at all.

Any insights as to what the configuration of the Telus AS852 GPON network
looks would be helpful. Or other observations in general on
technically-oriented persons who are doing similar with other ILECs.

Re: Ingress filtering on transits, peers, and IX ports

[Eric Kuhnke]

Aside from the BCPs currently being discussed for ingress filtering, I
would be very interested in seeing what this traffic looked like from the
perspective of your DNS servers' logs.

I assume you're talking about customer facing recursive/caching resolvers,
and not authoritative-only nameservers.

Considering that one can run an instance of an anycasted recursive
nameserver, under heavy load for a very large number of clients, on a $600
1U server these days... I wonder what exactly the threat model is.

Reverse amplification of DNS traffic returning to the spoofed IPs for DoS
purposes, such as to cause the nameserver to DoS a single /32 endpoint IP
being targeted, as in common online gaming disputes?

What volume of pps or Mbps would appear as spurious traffic as a result of
this attack?



On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG 
wrote:

> We recently received an email notice from a group of security
> researchers who are looking at the feasibility of attacks using spoofed
> traffic.  Their methodology, in broad strokes, was to send traffic to
> our DNS servers with a source IP that looked like it came from our
> network.  Their attacks were successful, and they included a summary of
> what they found.  So this message has started an internal conversation
> on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our
> customers, nor was it about BGP ingress filtering.  This tested our
> ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on
> our transit, IX, and peering links.  I have done this filtering in the
> enterprise world extensively, and it's very helpful to keep out bad
> traffic.  BCP 84 also discusses ingress filtering for SP's briefly and
> seems to advocate for it.
>
> We have about 15 IP blocks allocated to us.  With a network as small as
> ours, I chose to go with a static ACL approach to start the
> conversation.  I crafted a static ACL, blocking all ingress traffic
> sourced from any of our assigned IP ranges.  I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
> * Permit entries for IP assignments to customers running multi-homed BGP
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and
> your direct peering links?
> * How is it accomplished?  Through static ACL or some flavor of uRPF?
> * If you use static ACLs, what is the administrative overhead like?
> What makes it easy or difficult to update?
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian
>

Re: Residential GPON last mile for network engineers (Telus AS852 and others)

[Eric Kuhnke]

Very interesting. Looks like the intention is to bypass the ONT entirely
and use a GPON ONT SFP in ones own choice of small home router. If the ISP
wants to do some weird TR069 provisioning or other stuff it could be seen
as interfering with the proper management of their network if you remove
the CPE entirely.

In an ideal world, personally I would be totally fine with keeping a telco
provided small ONT configured as a dumb L2 bridge, with one optical
interface single strand (SC/APC) going to the ISP, and 1000BaseT to my own
router.

On Tue, Oct 13, 2020 at 6:51 PM Eric Dugas  wrote:

> I don't have any particular insights for Telus, but there is a huge thread
> about bypassing Bell ONTs on DSLReports:
> https://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC
> Cheers,
> Eric
> On Oct 13 2020, at 9:38 pm, Eric Kuhnke  wrote:
>
> With the growth of gigabit class single fiber GPON last mile services, I
> imagine a number of people reading the list must have subscribed to such by
> now.
>
> Something that I have observed, and shared observations with a number of
> colleagues, is that very often a person who works for ($someAS) lives in a
> location where you are effectively singlehomed to ($someotherAS). Maybe you
> bought your house before you got a job with your current employer, or maybe
> the network you work for doesn't do residential last mile service at all.
> Perhaps you work remotely for a regional sized entity that's a long
> distance away from where you live.
>
> Therefore necessitating a choice of service from whatever facilities based
> consumer-facing ISP happens to service your home.
>
> For example, in Seattle, a number of people discovered that they could
> keep the Centurylink GPON ONT, and remove the centurylink-provided
> router/modem combo device. Provided that they were able to configure their
> own router (small vyatta, pfsense box, mikrotik, whatever) to speak a
> certain VLAN tag on its WAN interface and be a normal PPPoE / DHCP client.
>
> I'm sure there are a lot of people who prefer to run their own home router
> and wifi devices, and not rely upon a ($big_residential_isp) provided
> all-in-one router/nat/wifi box with opaque configuration parameters, or no
> ability to change configuration at all.
>
> Any insights as to what the configuration of the Telus AS852 GPON network
> looks would be helpful. Or other observations in general on
> technically-oriented persons who are doing similar with other ILECs.
>
>

Re: Ingress filtering on transits, peers, and IX ports

[Eric Kuhnke]

If I had a dollar for every 'scary security alert' email received in a NOC
email inbox from a 'security researcher group' that is the results of a
port scan, or some small subset of trojan infected residential endpoint
computers attempting outbound connections on ($common_service_port), or
similar...



On Tue, Oct 13, 2020 at 7:50 PM Chris Adams  wrote:

> Once upon a time, Eric Kuhnke  said:
> > Considering that one can run an instance of an anycasted recursive
> > nameserver, under heavy load for a very large number of clients, on a
> $600
> > 1U server these days... I wonder what exactly the threat model is.
>
> A customer forwarded one of these notices to us - looked like it's about
> recursive DNS cache poisoning.  It's been a while since I looked
> closely, but I thought modern recursive DNS software was pretty
> resistant to that, and anyway, the real answer to that is DNSSEC.
>
> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
> from some group I've never heard of (and haven't AFAIK engaged the
> community about their "new" attack, scans, or notices)... seems more
> like shameless self promotion.
>
> --
> Chris Adams 
>

Re: Hurricane Electric AS6939

[Eric Kuhnke]

For small ISPs looking at setting up their first ever presence at an IX
point, you almost certainly would not be ordering an actual 'wave' (eg: a
specific DWDM channel on a legacy 10G DWDM platform, handed off to you with
1310/LX interfaces at both ends), but lit layer 2 transport service between
the carrier hotel and your service location.

Pricing for the two types of service can be quite different when you
request an actual 'wave' from a carrier sales person, vs just lit L2
transport capable of large MTUs, QinQ, etc.

The ISP carrying it might take it between those two places as simply a vlan
trunked through a larger 100G link, as a MPLS circuit, lots of possible
things.

Unless you happened to be in a happy conjunction of the right place at the
right time, and an older DWDM system on exactly the same path you wanted
happened to have an empty channel and ready to go interface cards at both
ends.






On Tue, Oct 13, 2020 at 11:12 PM Forrest Christian (List Account) <
li...@packetflux.com> wrote:

> Generally one would order a circuit (aka wave) between your location and
> the IX fabric at the interchange if you're not at the site you're wanting
> to peer at.
>
> For instance, the network I am the network engineer for has a circuit
> which terminates into the Seattle IX (SIX) fabric.   We don't have any
> other presence in Seattle (or Washington for that matter) at this point -
> our circuit connects directly to our port on the Exchange.   We're
> considering adding a similar link to another exchange point somewhere to
> the east or southeast of us.   I haven't looked at the graphs recently, but
> it's not uncommon for >50% of our traffic to come from the exchange.   And
> yes, we're peered with Hurricane and others there.
>
> We're also looking at dropping 1U or so of equipment in so we can pick up
> some transit as well, but that's a story for a different day about the joys
> of providing internet in the less populated parts of the country.
>
> In your case, it also looks like there are also some peering options at
> the datacenters you are currently at as well.   You may want to do some
> more research to determine how that might work in your situation.
>  PeeringDB is a good resource along with google searches for "peering 100
> Taylor" or "peering austin data foundry"
>
>
>
> On Tue, Oct 13, 2020 at 9:51 PM  wrote:
>
>> Don’t you have to be there to join?
>>
>>
>>
>> I’m in Austin and San Antonio
>>
>>
>>
>> -Aaron
>>
>>
>>
>> *From:* Mike Hammett 
>> *Sent:* Tuesday, October 13, 2020 7:20 PM
>> *To:* Aaron Gould 
>> *Cc:* nanog@nanog.org
>> *Subject:* Re: Hurricane Electric AS6939
>>
>>
>>
>> https://bgp.he.net/AS16527
>>
>>
>>
>> You don't appear to be on any IXes. Definitely join some IXes before
>> buying another 100G of transit.
>>
>>
>>
>> DFW has a couple and there are some more that are starting up.
>>
>>
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions 
>> 
>> 
>> 
>> 
>> Midwest Internet Exchange 
>> 
>> 
>> 
>> The Brothers WISP 
>> 
>> 
>> --
>>
>> *From: *"Aaron Gould" 
>> *To: *nanog@nanog.org
>> *Sent: *Tuesday, October 13, 2020 6:29:55 PM
>> *Subject: *Hurricane Electric AS6939
>>
>> Do y’all like HE for Internet uplink?  I’m thinking about using them for
>> 100gig in Texas.  It would be for my eyeballs ISP.  We currently have
>> Spectrum, Telia and Cogent.
>>
>> -Aaron
>>
>>
>>
>
>
> --
> - Forrest
>

Re: Hurricane Electric AS6939

[Eric Kuhnke]

The inverse of that is that an actual wavelength for 10/100G services can
be contractually defined to a certain specific path at OSI layer 1 (with
GIS vector shape files from the underlying carrier provided prior to
signing a contract). Whereas a layer 2 transport service could also turn
out to be unprotected, if it's a particularly low cost service.

Or it could be protected. And you might not have the ability to define its
path between city A and city B to intentionally avoid being non-diverse
from another route.

The L2 lit service carrier could re-route it around the region however they
want during the term of your service, if the contract isn't written to
avoid that.

In my experience actual wavelengths such as a carrier might use to
transport an STM64 between two places on far sides of a state, even
non-protected, will be considerably more expensive than buying lit L2
service. For small ISPs where their entire presence at an IX will fit in
one or two 10Gbps circuits, and a 100Gbps circuit from $smalltown to
$bigcity_ix_point would be cost prohibitive, it's often the best option.



On Wed, Oct 14, 2020 at 1:08 PM Darin Steffl 
wrote:

> The downside to waves are that they're typically not protected. So a cut
> will take you down. If you have 10G Layer 2 ethernet, they often will have
> redundant paths so the only single path that can fail is between you and
> their first POP where they hopefully have redundancy. It can make a big
> difference when you're transporting data hundreds or thousands of miles.
> The longer the path, the less reliable the wave will be as each route mile
> opens you up to more risk.
>
> On Wed, Oct 14, 2020 at 2:25 PM Mike Hammett  wrote:
>
>> I suppose it depends on your carrier and their capabilities.
>>
>> I much prefer waves to any kind of service that you can aggregate. Being
>> able to aggregate just means they're going to oversubscribe you and at some
>> point, you'll not get what you're paying for. Can't do that on a wave.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <https://www.facebook.com/mdwestix>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <https://www.facebook.com/thebrotherswisp>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> --
>> *From: *"Eric Kuhnke" 
>> *To: *"Forrest Christian (List Account)" 
>> *Cc: *"nanog list" 
>> *Sent: *Wednesday, October 14, 2020 2:25:46 AM
>> *Subject: *Re: Hurricane Electric AS6939
>>
>> For small ISPs looking at setting up their first ever presence at an IX
>> point, you almost certainly would not be ordering an actual 'wave' (eg: a
>> specific DWDM channel on a legacy 10G DWDM platform, handed off to you with
>> 1310/LX interfaces at both ends), but lit layer 2 transport service between
>> the carrier hotel and your service location.
>>
>> Pricing for the two types of service can be quite different when you
>> request an actual 'wave' from a carrier sales person, vs just lit L2
>> transport capable of large MTUs, QinQ, etc.
>>
>> The ISP carrying it might take it between those two places as simply a
>> vlan trunked through a larger 100G link, as a MPLS circuit, lots of
>> possible things.
>>
>> Unless you happened to be in a happy conjunction of the right place at
>> the right time, and an older DWDM system on exactly the same path you
>> wanted happened to have an empty channel and ready to go interface cards at
>> both ends.
>>
>>
>>
>>
>>
>>
>> On Tue, Oct 13, 2020 at 11:12 PM Forrest Christian (List Account) <
>> li...@packetflux.com> wrote:
>>
>>> Generally one would order a circuit (aka wave) between your location and
>>> the IX fabric at the interchange if you're not at the site you're wanting
>>> to peer at.
>>>
>>> For instance, the network I am the network engineer for has a circuit
>>> which terminates into the Seattle IX (SIX) fabric.   We don't have any
>>> other presence in Seattle (or Washington for that matter) at this point -
>>>

Re: Hurricane Electric AS6939

[Eric Kuhnke]

Yes it did, because they were running *all* of those over their Infinera
DWDM platforms which crashed. If the underlying optical line terminals are
FUBAR, all bets are off.



On Wed, Oct 14, 2020 at 2:27 PM Luke Guillory 
wrote:

> Didn’t the Dec 2018 CL outage cause waves and even TDM circuits to go down?
>
>
>
>
>
>
>
> Luke
>
>
>
>
>
>
>
> *From:* NANOG  *On
> Behalf Of *Matt Erculiani
> *Sent:* Wednesday, October 14, 2020 3:59 PM
> *To:* Darin Steffl 
> *Cc:* nanog list 
> *Subject:* Re: Hurricane Electric AS6939
>
>
>
> **External Email: Use Caution**
>
> For providers who use the same infrastructure for their IP backbone and
> Ethernet services (as so many do), a large DDoS could disrupt all Ethernet
> services that normally traverse affected links, whereas Waves would be
> blissfully ignorant of such an event. Waves are pretty reliable and will
> only go down as a result of a configuration error, vendor software issue,
> or physical/layer 1 failure, all of which can also affect Ethernet services.
>
>
>
> This is especially important if you select a provider that sees excess
> capacity as a wasted operational expense instead of an investment in
> reliability.
>
>
>
> Worth noting that protected Waves do have a "reconvergence" time like
> Ethernet would, but this is typically measured in nanoseconds for shorter
> distances. Your equipment can probably be configured to not link-down
> during this gap, you'll just see some errors or a few dropped packets
> (subject to your provider's specific implementation).
>
>
>
> -Matt
>
>
>
> On Wed, Oct 14, 2020 at 2:41 PM Darin Steffl 
> wrote:
>
> Yes but they're $$$ to have protection. Generally ethernet will be cheaper
> than waves with the added protection.
>
>
>
> I'm not arguing for one or the other. Waves will often be cheaper when
> looking at 10G or 100G compared to ethernet. For 1G or less, ethernet might
> be cheaper with some protection already built-in.
>
>
>
> On Wed, Oct 14, 2020 at 3:31 PM Mike Hammett  wrote:
>
> *nods* There are protected wave services generally available if you wish
> to protect about such things.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com/thebrotherswisp>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> --
>
> *From: *"Darin Steffl" 
> *To: *"Mike Hammett" 
> *Cc: *"Eric Kuhnke" , "nanog list"  >
> *Sent: *Wednesday, October 14, 2020 3:08:19 PM
> *Subject: *Re: Hurricane Electric AS6939
>
> The downside to waves are that they're typically not protected. So a cut
> will take you down. If you have 10G Layer 2 ethernet, they often will have
> redundant paths so the only single path that can fail is between you and
> their first POP where they hopefully have redundancy. It can make a big
> difference when you're transporting data hundreds or thousands of miles.
> The longer the path, the less reliable the wave will be as each route mile
> opens you up to more risk.
>
>
>
> On Wed, Oct 14, 2020 at 2:25 PM Mike Hammett  wrote:
>
> I suppose it depends on your carrier and their capabilities.
>
> I much prefer waves to any kind of service that you can aggregate. Being
> able to aggregate just means they're going to oversubscribe you and at some
> point, you'll not get what you're paying for. Can't do that on a wave.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <https://www.facebook.com/mdwestix>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <https://www.facebook.com

Re: Ingress filtering on transits, peers, and IX ports

[Eric Kuhnke]

I think he means packet captures from an example, voluntarily-tested
recursive nameserver subject to this attack.


On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio  wrote:

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway  wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks.  Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

Re: Linux router network cards

[Eric Kuhnke]

In addition to Jared's advice, I would recommend calculating PCI-Express
bandwidth bus points for whatever platform one is using.

For instance using the Intel X710-DA4, which could be capable in a maximal
scenario of 80Gbps of traffic, ensure it's in at least a PCI-E 3.0 x4 slot.
And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0 if a very new
system) lanes that exist and are connected to the CPU. Big difference
between some options for Ryzen and Threadripper vs Intel CPUs, towards the
lower end of the cost range.

With recent Linux kernels if you have an Intel 510 or 710 series two or
four port card in a slot that can't support its full capability, you'll get
a warning in dmesg at boot time.



On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger  wrote:

> I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based routing.
>
> If you're doing routing protocols, allocate 2 CPU cores to the control
> plane and then a CPU core per 10G/1G interface for the dataplane, plus an
> extra core for good measure. So for a 4 x 10G router taking in full routes,
> 2 cores for control plane, 5 cores for the dataplane. Those cores should be
> Intel Xeon E5-2600v3/4 or newer and faster the clocks, the better.
>
> Similar CPU core allocations if you choose TNSR.
>
> On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG 
> wrote:
>
>> Chelsio cards are probably what you are looking for.
>>
>> https://www.chelsio.com/terminator-6-asic/
>>
>> It's closer to an asic than a traditional nic as the router/firewall rules
>> are pushed directly into the hardware.
>>
>> I don't know how good they are with linux and they seem to be compatible.
>> https://www.chelsio.com/linux/
>>
>> You will need to mess around a bit and fiddle here and there. If you don't
>> mind using FreeBSD instead of linux, you could achieve a smoother and more
>> integrated experience.
>>
>> Jean
>>
>> -Original Message-
>> From: NANOG  On Behalf Of micah
>> anderson
>> Sent: Thursday, October 22, 2020 5:31 PM
>> To: Philip Loenneker ; NANOG
>> 
>> Subject: RE: Linux router network cards
>>
>>
>> Thanks for the reply.
>>
>> Philip Loenneker  writes:
>> > Take a look at the Mellanox ConnectX 5 series of cards. They handle
>> > DPDK, PVRDMA (basically SR-IOV that allows live migration between
>> > hosts), and can even process packets within the NIC for some
>>
>> From what I can tell, SR-IOV/PVRDMA aren't really useful for me in
>> building
>> a router that wont be doing any virtualization.
>>
>> If the card can do DPDK, can it do XDP?
>>
>> > The slidedeck for the presentation is here:
>> > https://www.ausnog.net/sites/default/files/ausnog-2019/presentations/1
>> > .9_Rhod_Brown_AusNOG2019.pdf
>> >
>> > It's heavily targeting virtualised workloads but some of the feature
>> sets
>> apply to bare-metal uses too.
>>
>> Yeah, this wont be a virtualized environment, just a router passing
>> packets,
>> dropping them, handling bgp and collecting flows.
>>
>> --
>> micah
>>
>>

Re: Linux router network cards

[Eric Kuhnke]

If building a lower end/low cost router this is absolutely a consideration.
In single socket regular ATX form factor, and products in the price range
of $165 for a motherboard and $250-400 price range for a CPU.

Comparing the PCI-E lanes available on an Intel Core i7 series to something
AMD zen/zen2 based (Ryzen), the AMD has greatly more. Some of the Intel
single socket core i5/i7 products have just enough PCI-E lanes for their
own onboard gigabit NIC and one PCI-E 3.0 x16 GPU for gaming purposes.

Would absolutely be a consideration if trying to build something with 8 to
12 10GbE interfaces capable of bursty traffic, but not flows and traffic
levels that would require line rate on all ports simultaneously.

On Sun, Oct 25, 2020 at 10:13 AM Vincent Bernat  wrote:

>  ❦ 24 octobre 2020 09:55 -06, Keith Medcalf:
>
> > And do not use an Intel CPU.
> >
> > Intel only has 4x PCIe lanes that are shared out into whatever
> > configuration they claim to have and are totally unsuitable for use in
> > a computer that actually has to be able to do high-speed I/O.
>
> That's likely to be incorrect. Intel CPU usually have 48 lanes for the
> Skylake generation. The 4 lanes limitation only applies to what is
> connected over DMI to the PCH, which is usually used for low-bandwidth
> stuff (1G NIC, SATA, 1x PCIe slots). Look at your motherboard manual to
> check how many lanes are affected to each component.
> --
> Make sure every module hides something.
> - The Elements of Programming Style (Kernighan & Plauger)
>

Re: cheap MPLS router recommendations

[Eric Kuhnke]

If we're talking about whitebox router and ipifusion, what we're really
talking about is vyatta/vyOS and the linux foundation DANOS stuff on an
ordinary x86-64 server that has a weird shape.

https://www.ipinfusion.com/commercial-version-of-danos-product-page/

https://www.danosproject.org/

In which case it really comes down to how comfortable you are with the
feature sets of the individual daemons contained within Vyatta/VyOS derived
products (FRR, etc), and then your trust level in the hardware. Typically
something such as a Taiwanese industrial/embedded platform manufacturer
such as Lanner:

http://www.lannerinc.com/products/network-appliances/x86-rackmount-network-appliances

If you look at the results of a linux kernel boot on a Lanner appliance
running VyOS, or a lspci -v, they're not significantly different than
taking a Dell or Supermicro rack server and sticking a whole bunch of Intel
or Chelsio 2 or 4-port 10GbE cards into it. It's just a weird shaped
motherboard, but ultimately derived from an Intel or AMD reference design,
and shares a lot in common in a block diagram with a 1U dual socket server
motherboard from a company like Tyan or Supermicro. You've got ethernet
NICs attached to the PCI-E bus the same as if they were slotted into cards.

Aside from the big names like Quanta, Compal and Clevo who will manufacture
these things for you in a bespoke fashion if you're a big cloud scale
operator, if you google "taiwan embedded industrial motherboard" you'll
find the companies that make most of the x86-64 whitebox router hardware.

I guess the point I'm trying to make above is that **if** you're confident
in both the SW and HW, you can disaggregate your choice of software
(vyatta/vyos/DANOS etc) from your own choice of hardware to best fit your
needs, rather than purchasing it together as a package.

On Wed, Oct 21, 2020 at 1:28 PM  wrote:

> Just to clarify what cheap means, ideally  -$2000 to $4000 new
>
> -new is preferred as buying used kit on second hand market one is at the
> mercy of the price fluctuations and availability.
>
>
>
> And the likes of the M2400 looks good 4x10G plus some 1G, unfortunately
> there are no details on the webpage (and the datasheet can’t be downloaded…
> )
>
>
>
> Are there more folks out there bundling open NOS and white-box HW along
> with the support for the whole thing?
>
>
>
>
>
> adam
>
>
>
> *From:* NANOG  *On
> Behalf Of *Colton Conor
> *Sent:* Monday, October 19, 2020 4:51 PM
> *To:* t...@pelican.org
> *Cc:* NANOG 
> *Subject:* Re: cheap MPLS router recommendations
>
>
>
> I haven't tried one myself, but Dasan Zhone has the M2400 and M3000.
> Basically, a whitebox with IP Infusion code on it. New, I think the price
> point is sub $2000 to $4000 new. That's a ton of ports for that price
> point. Anyone tried these yet?
> https://dzsi.com/product-category/mobile-xhaul/
>
>
>
>
>
> On Mon, Oct 19, 2020 at 3:38 AM t...@pelican.org  wrote:
>
> On Saturday, 17 October, 2020 00:41, "Tony Wicks"  said:
>
> > Well, there is always the MX104 (if you want redundancy) or MX80 if you
> > don’t. That will give you 80gig wire speed just don’t load it up with
> > more than one full table.
>
> Bear in mind that the MX80 is now in the EoL process, you have <4 years of
> support left.  Depending on your expected life-time / depreciation rules,
> buying one new right now might be unwise.
>
> Do *not* throw a full table at it (or any of the PowerPC Junipers) unless
> you have a lot of patience for reconvergence, and black-holes while you
> wait.
>
> MX104 is a nice box for getting dual-RE in something relatively compact
> and cheap, and has environmental hardening if that matters to you, but is
> still not best pleased with full tables.
>
> OP could do with clarifying "cheap" :)
>
> Regards,
> Tim.
>
>

dark fiber connection between 111 E 8th and Coresite NYC1 or NYC2

[Eric Germann]

Looking for a recommendation of a provider who can give us a dark fiber cross 
connect or an L2 connection between the two in the subject for an AWS Direct 
Connect out of Coresite

Thanks

Eric

Re: FCC Announces All Of Puerto Rico To Have Access To High-Speed Broadband Service

[Eric Kuhnke]

The press release doesn't reference at all, but Aeronet (the largest WISP
in Puerto Rico, and an operator of gigabit class service in MDUs) has been
testing Facebook/Terragraph 802.11ay 60 GHz based, point to multipoint last
mile stuff for a while now. Very short range, high speed, high capacity.

They use it in addition to a number of licensed band and 71-86 GHz PTP
links.

https://www.peeringdb.com/net/20459

Various 802.11ay based PtMP solutions are about to hit the market from 4 or
5 different competing vendors.





On Mon, Nov 2, 2020 at 8:22 AM Sean Donelan  wrote:

>
> FCC Announces All Of Puerto Rico To Have Access To High-Speed Broadband
> Service As A Result Of Uniendo A Puerto Rico Fund
>
> Nearly a Third of Locations Will Get Speeds of At Least 1 Gbps with All
> Other Locations Getting Speeds of At Least 100 Mbps
>
>
> https://www.fcc.gov/document/fcc-announces-usf-support-high-speed-broadband-puerto-rico
>
> WASHINGTON, November 2, 2020—The Federal Communications Commission’s
> Wireline Competition Bureau today announced that funding through Stage 2
> of the Uniendo a Puerto Rico Fund will result in all locations in Puerto
> Rico having access to fixed broadband service
> with speeds of at least 100 Mbps. And nearly one-third of those locations
> will have access to fixed broadband service with speeds of at least 1
> Gbps.
>
> Two winning applicants in the Uniendo a Puerto Rico Stage 2 Competitive
> Process submitted bids for $127.1 million in funding over 10 years
> covering more than 1.2 million locations through a competitive process
> that awarded support for fixed voice and broadband services based on the
> weighting of price and network performance, including speed, latency,
> usage allowance, and resiliency. Liberty Communications has committed to
> offering service to over 914,000 locations, and Puerto Rico Telephone
> Company will offer service to over 308,000 locations.
>

Re: Phoenix-IX Contact

[Eric Kuhnke]

Always a good time for network operators to consider the risks of having
any one person as a single point of failure for something kind of important:

https://en.wikipedia.org/wiki/Bus_factor

Disaster recovery and continuity of business plans should always include
the concept of what if some percentage of the key team members were to be
suddenly unavailable permanently (the Malaysian airline incident, for
example).



On Mon, Nov 9, 2020 at 8:08 PM Kate Gerry  wrote:

> Is there anybody else even there? I thought that it was all Paul's show!
>
> If I was able to (as in, had access to), I would offer to help/run with
> the IX. I may live in California, but it's a realistic car trip back and
> forth to Phoenix.
>
> --
> Kate
>
> On Nov 9, 2020, at 17:58, Mike Hammett  wrote:
>
> Paul's LinkedIn seems to show that he checked out in April. Let me know if
> you have any success reaching anyone there.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
> --
> *From: *"Kate Gerry" 
> *To: *"Bill Woodcock" 
> *Cc: *nanog@nanog.org
> *Sent: *Monday, November 9, 2020 5:44:42 PM
> *Subject: *Re: Phoenix-IX Contact
>
> Just a heads-up, I never heard a word from anybody at Phoenix-IX.
>
> Is there anybody still running the IX? Or is it just on autopilot? It'd be
> nice if anybody had some information on whatever happened to Paul.
> Hopefully he is okay!
>
> --
> Kate
>
> On Sep 14, 2020, at 13:33, Kate Gerry  wrote:
>
> Thank Bill! I've been trying to reach Paul for ages now, hopefully he pops
> back up again. We want to upgrade.
>
> On an unrelated note, it looks like somebody has their ticket system
> subscribed to the list... Awesome.
>
> *From: *Dating Support 
> *Subject: **[#WQV-291-95071]: Phoenix-IX Contact*
> *Date: *September 14, 2020 at 12:43:20 PDT
> *To: *kge...@outlook.com
> *Reply-To: *dating.supp...@csvwebsupport.com
>
> Kate Gerry,
>
> Thank you for contacting us. This is an automated response confirming the
> receipt of your ticket. Our team will get back to you within 24 hours.
>
>
> --
> Kate
>
>
> On Sep 14, 2020, at 12:48, Bill Woodcock  wrote:
>
>
>
> On Sep 14, 2020, at 9:31 PM, Kate Gerry  wrote:
>
> Does anybody have a contact who works at Phoenix-IX? I have been
> attempting to reach somebody there for a while now without any luck.
>
> Attempts to each out to peer...@phoenix-ix.net as well as Ninja-IX have
> been without any luck. We also tried reaching out to Paul Emmons via
> LinkedIn mail and never received a response.
>
>
> Paul is the correct person.
>
>-Bill
>
>
>

Re: Phoenix-IX Contact

[Eric Kuhnke]

I presume that the biggest telcos, cable MSOs and such in the Phoenix
region already operate PNIs with each other, so the real question would be
what population of ISPs and how much traffic would go across an IX if you
subtract the top-six largest last mile service providers.



On Tue, Nov 10, 2020 at 5:28 AM Walt  wrote:

> Is it time for a new IX in Phoenix?
>
>
>
> *From: *NANOG  on behalf of Kate
> Gerry 
> *Date: *Monday, November 9, 2020 at 11:07 PM
> *To: *Mike Hammett 
> *Cc: *"nanog@nanog.org" 
> *Subject: *Re: Phoenix-IX Contact
>
>
>
> Is there anybody else even there? I thought that it was all Paul's show!
>
>
>
> If I was able to (as in, had access to), I would offer to help/run with
> the IX. I may live in California, but it's a realistic car trip back and
> forth to Phoenix.
>
>
>
> --
>
> Kate
>
>
>
> On Nov 9, 2020, at 17:58, Mike Hammett  wrote:
>
>
>
> Paul's LinkedIn seems to show that he checked out in April. Let me know if
> you have any success reaching anyone there.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> [image: Image removed by sender.] [image:
> Image removed by sender.]
> [image:
> Image removed by sender.]
> [image:
> Image removed by sender.] 
> Midwest Internet Exchange 
> [image: Image removed by sender.] [image:
> Image removed by sender.]
> [image: Image
> removed by sender.] 
> The Brothers WISP 
> [image: Image removed by sender.]
> [image: Image removed by
> sender.] 
> --
>
> *From: *"Kate Gerry" 
> *To: *"Bill Woodcock" 
> *Cc: *nanog@nanog.org
> *Sent: *Monday, November 9, 2020 5:44:42 PM
> *Subject: *Re: Phoenix-IX Contact
>
> Just a heads-up, I never heard a word from anybody at Phoenix-IX.
>
>
>
> Is there anybody still running the IX? Or is it just on autopilot? It'd be
> nice if anybody had some information on whatever happened to Paul.
> Hopefully he is okay!
>
>
>
> --
>
> Kate
>
>
>
> On Sep 14, 2020, at 13:33, Kate Gerry  wrote:
>
>
>
> Thank Bill! I've been trying to reach Paul for ages now, hopefully he pops
> back up again. We want to upgrade.
>
>
>
> On an unrelated note, it looks like somebody has their ticket system
> subscribed to the list... Awesome.
>
>
>
> *From: *Dating Support 
>
> *Subject: [#WQV-291-95071]: Phoenix-IX Contact*
>
> *Date: *September 14, 2020 at 12:43:20 PDT
>
> *To: *kge...@outlook.com
>
> *Reply-To: *dating.supp...@csvwebsupport.com
>
>
>
> Kate Gerry,
>
> Thank you for contacting us. This is an automated response confirming the
> receipt of your ticket. Our team will get back to you within 24 hours.
>
>
>
> --
>
> Kate
>
>
>
>
>
> On Sep 14, 2020, at 12:48, Bill Woodcock  wrote:
>
>
>
>
>
> On Sep 14, 2020, at 9:31 PM, Kate Gerry  wrote:
>
> Does anybody have a contact who works at Phoenix-IX? I have been
> attempting to reach somebody there for a while now without any luck.
>
> Attempts to each out to peer...@phoenix-ix.net as well as Ninja-IX have
> been without any luck. We also tried reaching out to Paul Emmons via
> LinkedIn mail and never received a response.
>
>
> Paul is the correct person.
>
>-Bill
>
>
>

OpenNMS, openstreetmap, geocoding APIs and SNMP

[Eric Kuhnke]

Anyone that has used a recent version of OpenNMS has probably noticed that
the default home page view now includes an openstreetmap based view of
node/device status, by geographical location.

Section 18.3 here:
https://docs.opennms.org/opennms/releases/latest/guide-admin/guide-admin.html

https://wiki.opennms.org/wiki/Geographical_Maps

example image here:
https://www.tecmint.com/wp-content/uploads/2019/05/opennms-default-admin-dashboard.png

While customizing a newly setup opennms VM for a unique purpose, I began
thinking that there's two possible ways to go about accurately placing
nodes on such a map, on a very large scale, with scripting/automation in
the provisioning and monitoring process.

*Method 1 *(as opennms does now): As defined in RFC3418, put the standard
street address of the node in the SNMP sysLocation field. Typically in a
human readable text format that would be understood when posted to a
geolocation API SaaS.

Example being: 2200 6th Avenue, Seattle WA 98121

   sysLocation OBJECT-TYPE
   SYNTAX  DisplayString (SIZE (0..255))
   MAX-ACCESS  read-write
   STATUS  current
   DESCRIPTION
   "The physical location of this node (e.g., 'telephone
   closet, 3rd floor').  If the location is unknown, the
   value is the zero-length string."
   ::= { system 6 }

The above is dependent on a few things. First, every node needs to have a
standard format street address entered into its sysLocation field. But what
if you have a telecom tower site on a mountain or hillside? What if it's a
node in a developing nation environment where standard third-party
geolocation API providers (S.18.3 of the opennms documentation) can't
resolve anything? What if it's a device that moves around, like a router on
an ocean going vessel or aircraft?


*Method 2 *(backwards compatible, possibly an improvement): Treat the 255
character sysLocation field as a rudimentary three column CSV file with
pipe delimiters. Put the standard human readable description of the node
location in the first column. Populate the second column with the latitude
and longitude in WGS84 decimal degree format, such as:

Suite 402 2200 6th Ave, Seattle WA 98121|47.616380|-122.341673

Where both columns have six decimal places of precision
Column 2 is positive integer if north of the equator
Column 3 is negative integer if west of Greenwich.
Optional 4th column: Elevation in metres, MSL

In the method 2 example described above, opennms can be customized to
retrieve the lat/long from sysLocation and place nodes on the dashboard,
rendering the underlying map from a self-hosted openstreetmap intranet
server without needing to call an external geolocation API. Provided that
the data provisioned into the CPE/node in the first place is accurate (this
is absolutely a GIGO problem), node locations for difficult-to-geocode
install locations should become much more accurate.


*Some issues that may be encountered:*
Very old or improperly implemented snmpd on some devices does not support a
255 character field length, sometimes not even 100 characters. Varies
greatly by vendor implementation.

Provisioning and configuration system for nodes need an automated way to
enter those lat/long coordinates into columns 2 and 3 in a scripted,
automated way. In an example of small 1/10GbE capable business-class ISP
CPE demarc devices, *perhaps* as the result of a provisioning system's call
to a geolocation API. But with some sort of optional step for configuring a
node in a location that won't geocode - such as having a provisioning tech
manually scroll to and click a location on a map in a browser.

For ISPs with large numbers of CPE endpoint/stub nodes that don't presently
have valid sysLocation, it will be a manually laborious one-time process to
hunt down all extant CPEs and verify their street address vs customer
records, google maps/bing maps/openstreetmap, and manually edit the
sysLocation field.

Environments that would have many nodes expected to fail third-party
geolocation APIs, would probably want to customize some sort of
click-on-map GUI for the persons doing the provisioning and hand-off of
CPEs to field installation techs. Or possibly using GPS data acquired from
an installation technician's mobile phone (standard iOS and Android browser
APIs for requesting GPS data), as a custom data field in a ticketing
system. Data from that field in the ticketing system could then be fed to
another piece of software and pushed to a script to write the sysLocation.


*Possibly outside scope*:
It's possible to self host your own openstreetmap tile server. The vector
data for the entire world is freely available to download. The tile
rendering engine and all software needed to set it up is all open source.
This can be implemented for ISP management/monitoring networks (NOC map
dashboard functionality, etc) that for security reasons have no connection
whatsoever to the global routing table.

Nodes that m

Re: 95th billing and automation

[Eric Kuhnke]

 'cacti' isn't really a monolithic thing. Ultimately it's a gui front end
for rra files and rrdtool. If one chooses not to go down the route of disk
space intensive but lossless time series database interface metric storage
(influxdb or similar), we are talking about what level of detail is lost
over the week and month scales in an rra file.

The step settings and compression-over-time for the rra file, defined when
its first created, will affect precision as well as the poller interval. An
older default cacti install with 5 minute intervals and small rra files
(under 500KB max size for the whole time scale per single interface and
file) will look very different than on a cacti 1.2+ install set up for 1
minute poller intervals. If you do a new cacti install on Debian testing or
unstable, during the initial configuration process you'll have the options
to define these. The difference between the two is best seen on some sort
of circuit that sees lots of very brief bursty high Mbps/Gbps traffic.

In 2020 considering the low cost of disk space I would recommend something
time series database storage based, polled on 60s intervals for interface
bps. Even if each discrete 95th billed interface eats up several hundred MB
of disk space. You can always set up something later on, to truncate the
time series db to throw away all data older than 12 months.


On Thu, Dec 10, 2020, 10:29 AM Mehmet Akcin  wrote:

> hi there,
>
> i have asked about this in the past. What is the best tool out there to do
> 95th percentile billing. I have decided to use observium and librenms as
> result of responses but there seems to be some kind of billing module issue
> with these tools (thy are basically the same code).
>
> What are other systems besides observium and librenms (and old
> fashion cacti) people are using these days with 95th billing and
> integration with a CRM like salesforce/zoho, etc. I appreciate the
> responses.
>
> Mehmet
>

Are the days of the showpiece NOC office display gone forever?

[Eric Kuhnke]

With the covid19 situation, obviously lots of ISPs have their NOC personnel
working from home, with VPN (or remote desktop) access to all the internal
tools, VoIP at home, etc.

In the traditional sense, by "showpiece NOC" I mean a room designed for the
purpose of having large situational awareness displays on a wall, network
weathermaps and charts, alerting systems, composed of four or more big flat
panel displays. Ideally configured to be actually useful for NOC purposes
and also something impressive looking for customer tours.

To what extent potential customers find that sort of thing to be a
signifier of seriousness on the part of an ISP, I suppose depends on what
sort of customers they are, and their relative degree of technical
sophistication.

Are the days of such an environment gone forever?

Re: Are the days of the showpiece NOC office display gone forever?

[Eric Kuhnke]

I would not be surprised to see Cisco CTC (the alarm control
panel/monitoring software for 15454s carrying TDM, SDH/SONET circuits)
still being used in the year 2030 in some places.


On Wed, Dec 16, 2020 at 1:10 PM Paul Amaral  wrote:

> We used to have some CRTs with MRTG running in the late 90’s 😊
>
>
>
> P
>
>
>
> *From:* NANOG  *On Behalf Of *Eric
> Kuhnke
> *Sent:* Wednesday, December 16, 2020 3:50 PM
> *To:* nanog@nanog.org list 
> *Subject:* Are the days of the showpiece NOC office display gone forever?
>
>
>
> With the covid19 situation, obviously lots of ISPs have their NOC
> personnel working from home, with VPN (or remote desktop) access to all the
> internal tools, VoIP at home, etc.
>
>
>
> In the traditional sense, by "showpiece NOC" I mean a room designed for
> the purpose of having large situational awareness displays on a wall,
> network weathermaps and charts, alerting systems, composed of four or more
> big flat panel displays. Ideally configured to be actually useful for NOC
> purposes and also something impressive looking for customer tours.
>
>
>
> To what extent potential customers find that sort of thing to be a
> signifier of seriousness on the part of an ISP, I suppose depends on what
> sort of customers they are, and their relative degree of technical
> sophistication.
>
>
>
> Are the days of such an environment gone forever?
>
>
>
>
>
>
>

Re: Are the days of the showpiece NOC office display gone forever?

[Eric Kuhnke]

Perhaps I should have clarified: "from the perspective of persons who have
the word "Sales" in their job titles, considered to be impressive looking
for customer tours"


On Wed, Dec 16, 2020 at 4:25 PM Randy Bush  wrote:

> > In the traditional sense, by "showpiece NOC" I mean a room designed for
> the
> > purpose of having large situational awareness displays on a wall, network
> > weathermaps and charts, alerting systems, composed of four or more big
> flat
> > panel displays. Ideally configured to be actually useful for NOC purposes
> > and also something impressive looking for customer tours.
>
> though your message has a current date, its content seems to be at least
> 15 years old
>
> randy
>

Re: Nashville

[Eric Kuhnke]

>From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in
one place has a long recovery time when it's physically damaged. Something
to think about for entities that own and operate traditional telco COs and
their plans for disaster recovery.


Nv1

Here is the latest update:  6:46AM 12/27:

Work continues restoring service to the CRS routers in the Nashville
Central Office. One router remains out of service and the other is in
service with some links remaining out of service.

The working bridge will reconvene at 08:00 CT with the following action
plan:
Additional cabling added to the first portable generator to enable full
load capabilities (08:00 CT)
Pigtails with camlocks installed for easy swap; investigate possibility to
land generator on the emergency service board to give the site N+1 with a
manual ability to choose anyone. (08:00 CT)
check small power plants on floors 4 and 6 (08:00 CT)
Investigate water damage on 1st floor and energize if safe (08:00 CT
Air handlers for floors 4,5 and 6 (09:00 CT)
complete all transport work
Turn up SS7
Turn up 911 service - Approximately noon or after)
Turn up switching service.
TDM Switching team will reconvene at 09:00 CT and the Signaling team will
reconvene at 11:00 CT on 12/27/2020.
DMS equipment on the 1st floor will be assessed for water damage. Switching
teams will monitor power and HVAC restoration and will begin switch
restoration as soon as the go ahead is provided by the power team.

Recovery Priorities:
1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7
isolation & Datakit needed for Local Switch restoration). Transport SMEs
currently working to turn up transport equipment
2. 6th floor (ESINET Groomers)
3. 10th and 8th floors (N4E) – Trunks
4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS
5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks
6. 11th floor (DMS: 01T) – Trunks
7. 4th floor (STP and SCP with mates up in Donelson)

The next update will be issued at approximately 09:00 CT on December 27.



Nv2

As of 09:00 CT: Teams worked through the night to restore service and
improve conditions at the Nashville 2nd Ave Central Office. Since the
initial service impact, over 75% of the Out of Service Mobility Sites have
been restored. Certain call flows may be limited and should improve as
additional restoration activities complete.
The generator that is currently powering equipment on the 2nd and 3rd
floor, was refueled and ran with no issues through the night. Overnight,
the batteries connected to it, continued to charge. Teams have placed
additional power cables, which once connected, will allow the working
generator, to better handle the load in the building. In order to
accomplish this, the generator will need to be shut down for 15-30 minutes
this morning, so teams can connect the new cables to the system. The power
team reports they are still on target to restore power and cooling to the
5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will
be delivered this morning and strategically placed, in case it is needed to
assist in cooling the office.

There is a Call Center at 333 Commerce, in Nashville that does not have
network or phone services available. Corporate Real Estate (CRE) reports
there is some damage to that office, but the extent of the damage will not
be known until they can gain access to the site. Because of this, the
impacted Call Center ceased operations until further notice.

DMS switching equipment on the 1st floor will be assessed for water damage.
Switching teams will monitor power and HVAC restoration. Equipment power
ups will begin, as soon as the go ahead is provided by the power team.

Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT
Central Office providing critical communication for teams working
restoration efforts. There are 17 assets deployed in the field- 15 are on
air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in
hot-standby for FN Customers where macro service recently recovered. There
is 1 asset staged at a deployment site in KY where macro service restored,
and 8 additional assets are on route to Nashville today to fulfill pending
FN Customer requests. Incoming requests continue to be triaged. The ones in
areas where service looks to have been restored, are being held, while the
others are being prioritized to be dispatched upon.

The next update will be issued at approximately 14:00 CT, unless there is a
significant change in status.



Nv3

AT&T Nashville update below, received at 3:35PM 12/27.

Since the initial service impact, over 95% of the Out of Service Mobility
Sites have been restored. Certain call flows may be limited and should
improve as additional restoration activities complete.

Electricians have installed the additional power cables from the generator,
to the emergency bus. These new cables will allow the generator to support
more of the load, of the building. The portable chiller requested, has
a

Re: Where do your 911 fees go and why does 911 fail

[Eric Kuhnke]

The massive 911 failure in WA state a few years ago was ultimately caused
by a failure in CenturyLink/legacy qwest transport equipment, where the
PSAP register was physically located in Colorado and inaccessible from the
point of view of network equipment in WA.


On Tue, Dec 29, 2020, 1:19 PM Matt Erculiani  wrote:

> This isn't the place where state governments are looking for feedback, so
> surely this will fall on deaf ears, but...
>
> Who runs 911 services on top of a single carrier solution? I wouldn't run
> a 10 seat mom and pop outfit without at least a cellular backup on a
> different carrier.
>
> 911 services are certainly not treated as critical as the public is led to
> believe. Not that anyone here is surprised by this, but hopefully positive
> change can come out of this otherwise horrible event.
>
> -Matt
>
> On Tue, Dec 29, 2020 at 1:30 PM Sean Donelan  wrote:
>
>> The FCC published its annual report on state 911 fees
>>
>> https://www.fcc.gov/document/fcc-issues-annual-report-state-911-fees-1
>>
>> The report finds that in 2019, states and territories collected more than
>> $3 billion in 911 fees, and more than $200 million of that funding was
>> diverted for uses other than 911.
>>
>>
>> You can look up your individual state's 911 report here
>>
>> https://www.fcc.gov/general/911-fee-reports
>>
>>
>> In case you are interested in Tennessee's 911 service resiliancy:
>>
>> [...]
>> The project, referred to as NG911, involves utilization of the State’s
>> secure, private, outsourced Multiprotocol Label Switching (“MPLS”)
>> network
>> called “NetTN,” provided by AT&T and managed by Strategic Technology
>> Solutions (“STS”) in the Tennessee Department of Finance and
>> Administration. The new network improves redundancy, reliability, and 911
>> call delivery. It enhances interoperability and increases the ease of
>> communication between ECDs, allowing immediate transfer of 911 calls,
>> caller information, and other data on a statewide level. NG911 will also
>> provide alternate paths to process emergency calls in the event of an
>> outage, providing lifesaving capabilities in the event of an emergency
>> that would have been unachievable on the outdated analog network.
>>
>> [...]
>> In fiscal year 2019, the TECB spent $11,224,726 million implementing and
>> maintaining the NG911 project: $6,974,790 to integrate with and adapt the
>> Net TN system for NG911 purposes; $780,966 for non-recurring start-up
>> costs of the statewide hosted controller or Call Handling as a Service
>> program; $3,451,369 to maintain the twenty-four hour network operations
>> center to assist PSAPs with technical issues; and $17,600 for Esri GIS
>> software licensing.
>> [...]
>>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>

Just a heads up, apparently Ubiquiti had a breach.

[eric-list]

Official statement:
https://mailchi.mp/ubnt/account-notification?e=30527b2904

 

Sincerely,

 

Eric Tykwinski

TrueNet, Inc.

P: 610-429-8300

 

Re: Parler

[Eric Kuhnke]

Googling "Rob Monster Epik" will tell you just about everything you need to
know about that organization.

On Wed, Jan 13, 2021 at 3:42 PM Matt Corallo  wrote:

> In case anyone thought Amazon was being particularly *careful* around
> their enforcement of Parler's ban...this is from
> today on parler's new host:
>
> $ dig parler.com ns
> ...
> parler.com. 300 IN  NS  ns4.epik.com.
> parler.com. 300 IN  NS  ns3.epik.com.
> ...
> ns3.epik.com.   108450  IN  A   52.55.168.70
>
> $ whois 52.55.168.70
> ...
> OrgName:Amazon Technologies Inc.
>
> and for the curious, ns4.epik.com is hosted by an Epik sub, but from a
> cursory glance appears to be single-homed to
> CDN77, which is vaguely surprising to me.
>
> Matt
>
> On 1/10/21 3:23 AM, William Herrin wrote:
> > Anybody looking for a new customer opportunity? It seems Parler is in
> > search of a new service provider. Vendors need only provide all the
> > proprietary AWS APIs that Parler depends upon to function.
> >
> >
> https://www.washingtonpost.com/technology/2021/01/09/amazon-parler-suspension/
> >
> > Regards,
> > Bill HErrin
> >
>

Re: DoD IP Space

[Eric Kuhnke]

Organizations that I have seen doing as you describe, because they ran out
of RFC1918 IP space, are also often using their existing private IP space
wastefully in the first place. Rather than using DoD /8s internally, if
they absolutely need to support v4-only equipment on their internal
management networks, they might be better served by considering that maybe
every POP doesn't need its own /24.

I'm talking about things I've seen where all of the management/monitoring
IPs of the equipment at a site might fit very comfortably in a v4 /27. But
that would be a labor intensive IP space and management address auditing
process of renumbering things, fixing internal DNS and rDNS, and updating
any myriad of things that might have the direct IP addresses of stuff
hardcoded into configuration files.

Rather than doing all of the above, they simply go "hey here's a /8 that's
highly unlikely our management network will ever need to talk to it in a
global routing table", and continue on with their /24 plan per tiny POP.



On Wed, Jan 20, 2021 at 8:38 AM Dorn Hetzel  wrote:

> I am aware of some companies that have used parts of a DoD /8 internally
> to address devices in the field that are too old to ever support IPV6.
> Those devices also never interact with the public internet, and never will,
> so for them, I guess the only risk would be that some other internal system
> that wants to talk to those devices would not also be able to talk to any
> endpoint on the public internet that wound up using space allocated from
> that block, some time in the future.  Is that about right or am I missing
> some key failure point?
>
> On Wed, Jan 20, 2021 at 9:59 AM j k  wrote:
>
>> My question becomes, what level of risk are these companies taking on by
>> using the DoD ranges on their internal networks? And have they
>> quantified the costs of this outage against moving to IPv6?
>>
>> Joe Klein
>>
>> "inveniet viam, aut faciet" --- Seneca's Hercules Furens (Act II, Scene
>> 1)
>> "*I skate to where the puck is going to be, not to where it has been."
>> -- *Wayne Gretzky
>> "I never lose. I either win or learn" - Nelson Mandela
>>
>>
>> On Wed, Jan 20, 2021 at 9:06 AM John Curran  wrote:
>>
>>> Indeed.
>>> /John
>>>
>>> > On Jan 20, 2021, at 8:47 AM, Cynthia Revström  wrote:
>>> >
>>> > But if you do this, make sure you keep track of where you might have
>>> put policies like this in, in case the DoD sells some the space or whatever
>>> in the future.
>>>
>>>

Re: DoD IP Space

[Eric Kuhnke]

Additionally, examples of impersonating a corporate entity to acquire
unused IP space (Erie Forge and Steel's /16, anyone?) undoubtedly fall
under existing, pre-internet interstate commerce fraud laws...

http://web.mit.edu/net-security/Camp/2003/DBowie_IP_Hijacking.pdf

https://www.wired.com/images_blogs/threatlevel/files/edited-iphd-2.ppt



On Wed, Jan 20, 2021 at 9:54 AM John Curran  wrote:

> On 20 Jan 2021, at 12:17 PM, Bryan Fields  wrote:
>
>
> AFAIK IANA and the RIR's cannot enforce use of IP space assignments on any
> network.
>
>
>   While route hijacking isn't necessarily an ARIN issue, I will
> note that several US law enforcement agencies (FBI & NCIS Cybercrime units)
> are quite interested in such events and do investigate them looking for
> criminal activity.
>
> (See
> https://pc.nanog.org/static/published/meetings/NANOG77/2108/20191028_Elverson_Your_As_Is_v1.pdf
>  for
> details.)
>
> FYI,
> /John
>
> John Curran
> President and CEO
> American Registry for Internet Numbers
>
>

Re: Nice work Ron

[Eric Kuhnke]

> How many other Belize defuncts do they have?  How many offshore countries
like Belize are there in the region?

Based on my cursory knowledge of offshore corporate registrations in
Belize, Panama and the Cayman Islands, identifying those locations which
are only mailboxes versus actual business office addresses should not be
overly complicated or difficult.

In the era of Google Street View for most major urban areas the initial
search process can be done remotely, such as when it appears that dozens of
companies occupy one street address of a very small office building.

For instance look at the company registration offices, with hundreds of
corporate entities sharing one office suite address, which were created by
Mossack Fonseca in Panama City.

https://en.wikipedia.org/wiki/Mossack_Fonseca

The same principle would apply not just to LACNIC, but also to anybody who
wanted to go in detail through the number of ISPs and hosting companies
that nominally exist in Malta and Cyprus.


On Thu, Jan 21, 2021 at 10:25 AM Töma Gavrichenkov 
wrote:

> Peace,
>
> On Thu, Jan 21, 2021, 8:17 PM Jean St-Laurent via NANOG 
> wrote:
>
>>
>> https://krebsonsecurity.com/2021/01/ddos-guard-to-forfeit-internet-space-occupied-by-parler/
>>
>
> A disclaimer:
> - Standing for the sanity of the Internet routing;
> - Assuming (quite reliably) actual policy violation;
> - Assuming good faith
>
> — am I the only one to believe that (given that LACNIC had allocated an IP
> block to a company that doesn't conform to the LACNIC policies) what we
> urgently need to see next is the complete audit of the LACNIC operations,
> so that this doesn't look like selective enforcement?
>
> How many other Belize defuncts do they have?  How many offshore countries
> like Belize are there in the region?
>
> --
> Töma
>

Broadcom P2100G 100G PCI-E 4.0 interface and Linux

[Eric Kuhnke]

This might be a long shot, but if there is anyone out there with a system
that has one of these in it, running a very recent Linux kernel:

https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g

I'm looking for a copy of the output from 'dmesg' on boot and output from
"lspci -v" to see exactly how it shows up on the PCI-E bus.

Starlink terminal data acquisition for network engineers

[Eric Kuhnke]

I thought about posting this to only NANOG, but since a great concentration
of beta testers of a technical/network engineering inclination are located
in the Pacific NW, decided to also include the SIX chat list.

You may have seen the Starlink android or ios consumer-friendly app, which
displays network traffic, uptime/downtime, and other link stats. I believe
this to be polled directly from the antenna unit itself over grpc.

The beta antennas are always 192.168.100.1. If you are using your own
router with the starlink beta system, in addition to its WAN interface
being an ordinary DHCP client in cgnat IP space, you'll need to manually
give it an address in that /24 and set up routing to reach the .1 IP as
needed.

reference: https://github.com/sparky8512/starlink-grpc-tools

reference: https://github.com/fullstorydev/grpcui

you'll need a fairly normal Linux or BSD box with:

git
go
python3
pip

use pip to install grpcio and grpcio-tools

install grpcurl: https://github.com/fullstorydev/grpcurl

do a git clone of the starlink-grpc-tools url above, also take a look at
its readme info

get the dish's protoset file and write it to new file dish.protoset , this
is an index of all data that can be polled
cd /home/eric/starlink-grpc-tools
/home/eric/go/bin/grpcurl -plaintext \
-protoset-out dish.protoset \
192.168.100.1:9200 \
describe SpaceX.API.Device.Device

/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"get_history\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle | python parseJsonHistory.py

output of the above looks like this:
2021-02-06T08:15:56,3600,19.92034332,14,2,0.3125,0,0,0.0,0

full CSV header for the above:
datetimestamp_utc,samples,total_ping_drop,count_full_ping_drop,count_obstructed,total_obstructed_ping_drop,count_full_obstructed_ping_drop,count_unscheduled,total_unscheduled_ping_drop,count_full_unscheduled_ping_drop

since we are able to acquire the above in a comma-delimited csv format,
it's fairly easy to write a script storing the integers from any one of
those particular columns into a mariadb db, sqlite, influxdb, or whatever.

the following will output about 3.8MB of text for the full history (I
believe this to be the full copy of the ring buffer stored in RAM for the
terminal's statistics) , pipe it into a text file if you want to manually
look at it.

/home/eric/go/bin/grpcurl -plaintext -d {\"get_history\":{}}
192.168.100.1:9200 SpaceX.API.Device.Device/Handle


same as the above but human readable output

/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"get_history\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle | python parseJsonHistory.py -v

current counter:   299673
All samples:   43200
Valid samples: 43200
Parsed samples:3600
Total ping drop:   20.03700998
Count of drop == 1:14
Obstructed:2
Obstructed ping drop:  0.3125
Obstructed drop == 1:  0
Unscheduled:   0
Unscheduled ping drop: 0.0
Unscheduled drop == 1: 0


see the get_history_notes.txt file for more info


SOME EXAMPLE QUERIES
these should match with what the json query is in the grpc GUI
# get status
/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"getStatus\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle

# get device info
/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"getDeviceInfo\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle

# get history, this outputs a huge amount of data
/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"getHistory\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle


The following is copied/pasted from my notes file on things we can acquire,
and then use a tiny shell script with awk, sed, regex, whatever as needed
to trim out just the numbers, and put them somewhere else for polling by
snmp extend

/home/eric/go/bin/grpcurl \
-plaintext \
-d {\"getStatus\":{}} \
192.168.100.1:9200 \
SpaceX.API.Device.Device/Handle

notes on what's what:
figures we care about to parse out and turn into just the integers
snr can never be higher than 9
fractionobstructed appears to be the percentage of the time that the view
is obstructed, as long as the view
remains unobstructed, this number appears to slowly decrement over time
validS is valid seconds? i think
the S is almost likely always Seconds of time

"uptimeS": "304439"
"snr": 9,
"fractionObstructed": 0.0013524292,
"validS": 61815.74,
"last24hObstructedS": 53
"downlinkThroughputBps": 47915.73,
"uplinkThroughputBps": 34980.496,
"popPingLatencyMs": 29.26


example of running the command above twice, the second time a few minutes
after the first,
to see that fractionObstructed does decrement itself over time
first run: 0.0013524292
second run: 0.0013467998

Re: Problems with newish IP block assignment issues from ARIN

[Eric Kuhnke]

One common cause of this issue is entities out there that have very old
'bogons' filters in place for the larger block, as an entire /8, /12 to /16
size of space that, many years ago, was unallocated space. Without getting
the end point organizations running the httpd, firewalls or whatever to fix
their broken configuration, it's a hard issue to fix from your end.

On a longer term time scale like multiple years, the reachability of an IP
block like yours will gradually increase as people with broken services are
contacted by additional persons to say "hey, this really is valid ARIN IP
space".



On Mon, Feb 8, 2021 at 12:15 PM Justin Wilson (Lists) 
wrote:

> Folks,
> Have a gremlin we have been chasing around for several months now and it’s
> becoming a major issue as we are getting tighter on IPV4 and needing to
> give some provider assigned space back.
>
> In June we received a /22 from ARIN.  As is my workflow I started
> announcing it but waited a month while I checked out the geolocation
> databases for correct info, did testing ,etc. All this time our test
> accounts could browse web-sites, etc.
>
> We put one of the pools into production and things ran good for awhile.
> Then we started getting the occasional web-site was not working.  After
> several of these we started assigning the customer an IP out of one of our
> other ARIN blocks and the web-site would be fine and reachable. The issue
> seems to reside just on this /22.  We have other blocks from ARIN and they
> are just fine.  We can assign an IP out of this new block and can’t reach
> certain web-sites.  We turn around and assign out of another block and
> web-site works just fine.
>
> We have two upstreams and an IX on this network.  We have tried
> withdrawing the route on this particular /22 and isolating to one upstream
> alone and the problems still persist.
>
> Many of the web-sites in question are government (both state and local),
> online universities, and the occasional local news station.  They are
> diverse enough to not be traced down to a common point, except the IP
> block.
>
> We announce the IP block via BGP the same exact way we announce the other
> blocks. Traceroutes show the path going the same way no matter what IP
> block the customer has.
>
> It acts like the IP block was blacklisted at some point and got on some
> bad lists but I don’t want ti limit myself to that theory.  I have opened
> up a ticket with ARIN asking for any guidance.  Has anyone ran into this
> with new space assigned? Any tools, sites, etc. I can use to do further
> troubleshooting.  The IP block does not appear to have any blacklisted IPs
> according to MX toolbox, and some others.
>
> The block in question is 134.195.44.0/22.  It has been RPKI certified and
> has IRR entries.
>
> Thanks in advance
>
>
> Justin Wilson
> j...@mtin.net
>
> —
> https://j2sw.com - All things jsw (AS209109)
> https://blog.j2sw.com - Podcast and Blog
>
>

Re: DoD IP Space

[Eric Kuhnke]

You don't, you wastefully assign a /24 to every unique thing that you think
needs an internal management IP block (even if there's 5 things that answer
pings there), and decide it's too much work to renumber things. Easy for a
big ISP that's also acquired many small/mid-sized ISPs to run out of v4
private IP space that way.



On Wed, Feb 10, 2021 at 4:05 AM Owen DeLong  wrote:

> Please explain to me how you uniquely number 40M endpoints with RFC-1918
> without running out of
> addresses and without creating partitioned networks.
>
> If you can’t, then I’m not the one making excuses.
>
> Owen
>
>
> > On Feb 9, 2021, at 15:44 , Izaac  wrote:
> >
> > On Fri, Feb 05, 2021 at 02:36:57PM -0800, Owen DeLong wrote:
> >> it is definitely possible to run out of RFC-1918 space with scale and
> no incompetence.
> >
> > No, it isn't.  It's the year 2021.  Stop making excuses.
> >
> > --
> > . ___ ___  .   .  ___
> > .  \/  |\  |\ \
> > .  _\_ /__ |-\ |-\ \__
>
>

RIPE Atlas probe available on SpaceX Starlink beta terminal

[Eric Kuhnke]

https://atlas.ripe.net/probes/1001821/

I am running what I believe to be the first RIPE Atlas probe on a Starlink
beta test terminal.

When searching the index of public probes I did not find any other probes
with "spacex" or "starlink" in the descriptions.

This probe is at present not contained within AS14593 (Starlink). All beta
test terminals that I am aware of right now, including my own, are in cgnat
IP space and meet the public Internet via AS36492 (Google).

This particular terminal is topologically closest to things at major IX
points in the metro Seattle area. The absolute lowest ping time I've seen
to something at the Westin is 15.85ms, with averages more often between
21-32ms.

The site where this is installed will occasionally conduct other, non Atlas
related tests which result in full saturation of the upstream or downstream
connection, simulating a number of different types of real world use cases.
This means that occasionally the RTT ping to things in Seattle may rise as
high as 150ms, and artificially-induced packet loss will be introduced into
the connection.

Please feel free to contact me off-list with any questions.

Infomart Dallas is on generator

[Eric Kuhnke]

I have now heard from two reliable sources that Infomart Dallas is
presently on generator, and is likely to remain so until the cold
weather/electrical supply emergency in Texas has abated. No network impact
seen yet.

Re: Infomart Dallas is on generator

[Eric Kuhnke]

http://www.ercot.com/

The 501c(4) nonprofit entity which controls the Texas grid. They've been
publishing load shedding updates.

On Mon, Feb 15, 2021, 5:07 PM Randy Bush  wrote:

> > From the latest update it sounds like rolling power outages in Dallas as
> > most places in Texas
>
>
> https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/
>

Re: Texas internet connectivity declining due to blackouts

[Eric Kuhnke]

See also, regional maps here. Thanks to CAIDA and the IODA project.

https://ioda.caida.org/ioda/dashboard

On Mon, Feb 15, 2021, 5:54 PM Sean Donelan  wrote:

> Not as bad as Myanmar (14%), Internet connectivity in Texas has been
> declining today.  According to NetBlocks, which normally monitors
> government imposed outages, reports network connectivity at 68% in Texas.
>
> https://netblocks.org/
>
> Texas operates a separate electric grid, with limited interconnections
> to the rest of North America.  For political reasons
>
> For those with long memories, ENRON a Texas based corporation, once upon a
> time drove rolling blackouts across California in order to make billions.
>

Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Eric Kuhnke]

That depends on your definition of grey market, there is an officially
approved ARIN IP block transfer process for people who are buying, via
brokers, discrete /24s and larger.



On Tue, Feb 16, 2021, 4:46 PM Michael Thomas  wrote:

>
> On 2/16/21 4:18 PM, Fred Baker wrote:
> > You may find this article interesting:
> >
> https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/
> > <
> https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/
> >
> >
> So aside from Afrinic, this is all being done on the gray market?
> Wouldn't you expect that price to follow something like an exponential
> curve as available addresses become more and more scarce and unavailable
> for essentially any price?
>
> Mike
>
>
> > Sent from my iPad
> >
> >> On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:
> >>
> >> 
> >> Basically are there places that you can't get allocations? If so,
> >> what is happening?
> >>
> >> Mike
> >>
>

Re: Viable Third Option?

[Eric Kuhnke]

In the context of Montreal, to clarify, when you say Zayo are you referring
to Zayo Canada (former AT&T Canada/MTS-Allstream), or AS6461, the original
Abovenet AS which is Zayo USA's IP transit network?


On Wed, Feb 17, 2021 at 11:17 AM Eric Dugas via NANOG 
wrote:

> The details you mentioned about Cogent and HE are still right.
>
> I'm managing an eyeball network and have Cogent in our blend but I also
> have three other Tier1s and VERY extensive peering (public and private). We
> have (from the cheapest to most expensive) Cogent, Telia, Zayo and Tata. I
> have to mention that we're based in Montreal so less choices compared to
> your market. The only two other Tier1s available in Montrea is GTT and
> Lumen/CL/Level3.
>
> Cogent: difficult relations, good service overall
> Telia: excellent relations, good service overall
> Tata: good relations, good service overall
> Zayo: difficult relations, good service overall
>
> Eric
>
> On Feb 17 2021, at 1:49 pm, Mike Hammett  wrote:
>
> This is from the perspective of an eyeball network. I understand that
> content networks would have different objectives and reasons. For instance,
> I have little to no reason as an eyeball network to exchange traffic with
> any other eyeball network (aside from P2P games). For a content network,
> getting into the eyeball networks is their objective.
>
> My crystal ball tells me this thread will spiral out of control because
> people won't be able to keep it on topic, but it is a question that I hear
> VERY often. I also expect a lot of purely bad or outdated information to
> get thrown out.
>
> Please try to keep it on topic and not being pedantic over relatively
> unimportant details.
>
> There are two major low-cost providers, Cogent and HE.
>
> Cogent
>
>- Refuses to peer IPv6 with HE
>- Refuses to peer IPv6 with Google
>- Aggressive sales tactics
>
> Hurricane
>
>- Doesn't have Cogent IPv6 because of Cogent's refusal
>- Lack of communities for anything other than blackholes
>
>
> I know there are a variety of other providers such as Fusion Network that
> operate at similar price points, but are available in way fewer locations.
>
> What else is out there? Anyone else that isn't 5x, 10x the cost?
>
> Cogent and HE get looked down upon (and sometimes deservedly so), but when
> I talk to someone trying to sell me a port in 350 Cermak for 8x the cost of
> Cogent and HE, you better have a very good argument for why you're worth
> it...  and they never do. "We're not Cogent." "and?" Many times I'm quoted
> transit that costs more than Cogent + IX + HE and they don't really have a
> good argument for it.
>
> As an eyeball, I join an IX and there goes 50% - 85% of my traffic and
> almost all of my traffic that anyone is going to notice or complain about
> if there are issues (video streaming).
>
> I do understand that enterprise eyeballs may have different requirements.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>
>

Re: Famous operational issues

[Eric Kuhnke]

On that note, I'd be very interested in hearing stories of actual incidents
that are the cause of why cardboard boxes are banned in many facilities,
due to loose particulate matter getting into the air and setting off very
sensitive fire detection systems.

Or maybe it's more mundane and 99% of the reason is people unpack stuff and
don't always clean up properly after themselves.

On Wed, Feb 17, 2021, 6:21 PM Owen DeLong  wrote:

> Stolen isn’t nearly as exciting as what happens when your (used) 6509
> arrives and
> gets installed and operational before anyone realizes that the conductive
> packing
> peanuts that it was packed in have managed to work their way into various
> midplane
> connectors. Several hours later someone notices that the box is quite
> literally
> smoldering in the colo and the resulting combination of panic, fire drill,
> and
> management antics that ensue.
>
> Owen
>
>
> > On Feb 16, 2021, at 2:08 PM, Jared Mauch  wrote:
> >
> > I was thinking about how we need a war stories nanog track. My favorite
> was being on call when the router was stolen.
> >
> > Sent from my TI-99/4a
> >
> >> On Feb 16, 2021, at 2:40 PM, John Kristoff  wrote:
> >>
> >> Friends,
> >>
> >> I'd like to start a thread about the most famous and widespread Internet
> >> operational issues, outages or implementation incompatibilities you
> >> have seen.
> >>
> >> Which examples would make up your top three?
> >>
> >> To get things started, I'd suggest the AS 7007 event is perhaps  the
> >> most notorious and likely to top many lists including mine.  So if
> >> that is one for you I'm asking for just two more.
> >>
> >> I'm particularly interested in this as the first step in developing a
> >> future NANOG session.  I'd be particularly interested in any issues
> >> that also identify key individuals that might still be around and
> >> interested in participating in a retrospective.  I already have someone
> >> that is willing to talk about AS 7007, which shouldn't be hard to guess
> >> who.
> >>
> >> Thanks in advance for your suggestions,
> >>
> >> John
>
>

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Eric Kuhnke]

There is really no such thing since there is just the one cable landing
station. I've previously spent months working in network infrastructure and
telecom in Sierra Leone, contact me off-list if you're serious about
getting something done there.




On Thu, Feb 18, 2021 at 9:46 AM Rod Beck 
wrote:

> Every time I try to bring a circuit into Africa it is like a complete tour
> of Dante's Hell.
>
> 😃
>
> Regards,
>
> Roderick.
>
> Roderick Beck
> Global Network Capacity Procurement
>
> United Cable Company
> www.unitedcablecompany.com
> https://unitedcablecompany.com/video/
> New York City & Budapest
>
> rod.b...@unitedcablecompany.com
>
> Budapest: 36-70-605-5144
>
> NJ: 908-452-8183
>
>
>
> [image: 1467221477350_image005.png]
>

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Eric Kuhnke]

Sierra Leone is very much *not* French speaking, in the context of ISPs and
telecom.

There may be a significant minority of people who do speak French due to
its regional proximity to other countries, for business, but the language
of higher education, business, finance, telecom, real estate and so forth
is all in English.

On Fri, Feb 19, 2021 at 3:20 AM Rod Beck 
wrote:

> I am sure South Africa is better. I am really referring to French speaking
> Western Africa.
>
> -R.
>
> --
> *From:* NANOG 
> on behalf of Mark Tinka 
> *Sent:* Friday, February 19, 2021 5:09 AM
> *To:* nanog@nanog.org 
> *Subject:* Re: Carrier Neutral Site - Freetown, Sierra Leone?
>
>
>
> On 2/18/21 19:45, Rod Beck wrote:
>
> Every time I try to bring a circuit into Africa it is like a complete tour
> of Dante's Hell.
>
>
> A broad brush for such a large place.
>
> Mark.
>

Re: Famous operational issues

[Eric Kuhnke]

>From a datacenter ROI and economics, cooling, HVAC perspective that might
just be the best colo customer ever. As long as they're paying full price
for the cabinet and nothing is *dangerous* about how they've hung the 2U
server vertically, using up all that space for just one thing has to be a
lot better than a customer that makes full and efficient use of space and
all the amperage allotted to them.


On Thu, Feb 18, 2021 at 11:38 AM t...@pelican.org  wrote:

> On Thursday, 18 February, 2021 16:23, "Seth Mattinen" 
> said:
>
> > I had a customer that tried to stack their servers - no rails except the
> > bottom most one - using 2x4's between each server. Up until then I
> > hadn't imagined anyone would want to fill their cabinet with wood, so I
> > made a rule to ban wood and anything tangentially related (cardboard,
> > paper, plastic, etc.). Easier to just ban all things. Fire reasons too
> > but mainly I thought a cabinet full of wood was too stupid to allow.
>
> On the "stupid racking" front, I give you most of a rack dedicated to a
> single server.  Not all that high a server, maybe 2U or so, but *way* too
> deep for the rack, so it had been installed vertically.  By looping some
> fairly hefty chain through the handles on either side of the front of the
> chassis, and then bolting the four chain ends to the four rack posts.  I
> wish I'd kept pictures of that one.  Not flammable, but a serious WTF
> moment.
>
> Cheers,
> Tim.
>
>
>

Re: Famous operational issues

[Eric Kuhnke]

I would be more interested in seeing someone who HASN'T crashed a Cisco
6500/7600, particularly one with a long uptime, by typing in a supposedly
harmless 'show' command.


On Tue, Feb 23, 2021 at 2:26 PM Justin Streiner  wrote:

> An interesting sub-thread to this could be:
>
> Have you ever unintentionally crashed a device by running a perfectly
> innocuous command?
> 1. Crashed a 6500/Sup2 by typing "show ip dhcp binding".
> 2. "clear interface XXX" on a Nexus 7K triggered a cascading/undocument
> Sev1 bug that caused two linecards to crash and reload, and take down about
> two dozen buildings on campus at the .edu where I used to work.
> 3. For those that ever had the misfortune of using early versions of the
> "bcc" command shell* on Bay Networks routers, which was intended to make
> the CLI make look and feel more like a Cisco router, you have my
> condolences.  One would reasonably expect "delete ?" to respond with a list
> of valid arguments for that command.  Instead, it deleted, well...
> everything, and prompted an on-site restore/reboot.
>
> BCC originally stood for "Bay Command Console", but we joked that it
> really stood for "Blatant Cisco Clone".
>
> On Tue, Feb 16, 2021 at 2:37 PM John Kristoff  wrote:
>
>> Friends,
>>
>> I'd like to start a thread about the most famous and widespread Internet
>> operational issues, outages or implementation incompatibilities you
>> have seen.
>>
>> Which examples would make up your top three?
>>
>> To get things started, I'd suggest the AS 7007 event is perhaps  the
>> most notorious and likely to top many lists including mine.  So if
>> that is one for you I'm asking for just two more.
>>
>> I'm particularly interested in this as the first step in developing a
>> future NANOG session.  I'd be particularly interested in any issues
>> that also identify key individuals that might still be around and
>> interested in participating in a retrospective.  I already have someone
>> that is willing to talk about AS 7007, which shouldn't be hard to guess
>> who.
>>
>> Thanks in advance for your suggestions,
>>
>> John
>>
>

Is there an established method for reporting/getting removed a company with 100% false peeringdb entries?

[Eric Kuhnke]

First, take a look at this:

https://www.peeringdb.com/asn/18894


Now look at these (or use your own BGP table analysis tools):

https://bgp.he.net/AS18894

https://stat.ripe.net/18894

The claimed prefixes announced, traffic levels and POPs appear to have no
correlation with reality in global v4/v6 BGP tables.

It is also noteworthy that I have inquired with a number of persons I know
who are active in network engineering in NYC, and nobody has ever
encountered this company.

Re: DPDK and energy efficiency

[Eric Kuhnke]

A great deal of this discussion could be resolved by the use of a $20
in-line 120VAC watt meter [1] plugged into something as simple as a $500 1U
server with some of the DPDK-enabled network cards connected to its PCI-E
bus, running DANOS.

Characterizing the idle load, average usage load, and absolute maximum
wattage load of an x86-64 platform is excessively difficult or complicated.

[1]
https://www.homedepot.com/p/Kill-A-Watt-Electricity-Monitor-P4400/202196386


On Thu, Mar 4, 2021 at 11:28 AM Etienne-Victor Depasquale 
wrote:

> *TL;DR - DPDK applications embody the phrase caveat emptor.*
>
> As Robert Bays put it:  "Please ask your open source dev and/or vendor of
> choice to verify."
> On the other hand, I do not recommend taking the following (citing Robert
> Bays again) for granted:
> "But the reality is [open source projects and commercial products] have
> all been designed from day one not to unnecessarily consume power."
>
> This note is presented in two sections.
> Section 1 presents the preamble necessary to avoid misinformation.
> Section 2 presents the survey.
>
> If so inclined, please read on.
>
> *SECTION 1*
> There are three issues at stake:
>
> 1.  the ground truth about the power/energy efficiency of (current)
> deployments that use DPDK,
> 2.  my choice of words for the first question, as this constitutes the
> claimed source of misinformation, and
> 3.  apportionment of responsibility for the attained level of
> power/energy efficiency of a deployment that uses DPDK,
>
> *Issue #1: ground truth on current deployments*
> I base on (a) research papers and (b) Pawel Malachowski's data. Numbered
> references are listed at the end of this e-mail.
>
> [1] investigates software data planes, including OvS-DPDK. Citing directly:
> "DPDK-OVS always works with high power consumption even when [there is] no
> traffic to handle.
> Considering the inefficiency [][in] power, DPDK provides power management
> APIs to compromise
> between power consumption and performance."
> "For DPDK-OVS, due to the feature of DPDK’s Polling Mode Driver (PMD),
> once the first DPDK port is added to vswitchd process,
> it creates a polling thread and polls DPDK device in continuous loop.
> Therefore CPU utilization for that thread is always 100%,
> and the power consumption r[]ises to about 138 Watt"
>
> [2] investigates multimedia content delivery and benchmarks *DPDK-OvS* in
> the process. Citing directly:
> "Even when no traffic was in transit,
> OvS-DPDK consumed approximately three
> times more energy than the other two data
> planes, adding 250 percent energy overhead
> (15.57 W) on top of the host OS."
>
> [3] proposes the use of ACPI P-states and the halt instruction to control
> power consumption,
> in the context of *a bespoke application*. Citing directly:
> "For example, a Xeon(R) E5-2620 v3 dual socket CPU consumes
> about 22W of power when it is idle; but if a DPDK-based software
> router runs on it, the CPU power soars to 83W even
> when no packets arrive. That is a power gap of more than
> 60W."
>
> [4] investigates the energy-efficient use of *Pktgen-DPDK*. Citing
> directly:
> "We find that high performance comes at the cost of high energy
> consumption."
>
> Pawel Malachowski  shows a list of cores (13 out of 16) in use by a DPDK
> application
> ("DPDK-based 100G DDoS scrubber currently lifting some low traffic using
> cores 1-13
> on 16 core host. It uses naive DPDK::rte_pause() throttling to enter C1").
> The list shows the cores spending most of their time in C1.
> This means that cores are in a low-power-idle state and therefore not in
> an active (C0) state.
> This shows a power-aware DPDK application.
>
> *Issue #2: my choice of words, as a source of misinformation*
> Issue has been taken with the text of question 1.
> I addressed this to the NANOG community,
> who are busy and knowledgeable.
> I chose, *with hindsight wrongly*, to paraphrase,
> with the expectation that a reader would interpret correctly.
> A better expression, that would still have been terse, would be:
> "Are you aware that *naïve* use of DPDK on a processor core keeps
> utilization at 100% regardless of packet activity?"
>
>
> *Issue #3: apportionment of responsibility for the attained level of
> power/energy efficiency of a deployment that uses DPDK*
> Pawel Malachowski states that "It consumes 100% only if you busy poll
> (which is the default approach)."
>
> Since it is the application that exploits the DPDK API,
> and since the DPDK API promotes run-to-completion (
> https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html),
> then *it is the application that determines power consumption*
> but it is DPDK's poll-mode driver *that poses a real threat to power
> efficiency, if used in "the default approach".*
>
> Robert Bays states:
> "The vast majority of applications that this audience would actually
> install in their networks do not do tight polling all the time and
> therefore don’t consume 100% of the CPU all th

Re: DPDK and energy efficiency

[Eric Kuhnke]

That was an unfortunate typo on my part, I meant to write "isn't
excessively difficult..."

Some real world examples of specific models of CPU + motherboard + PCI-E
NIC combinations with wattage figures at idle load, average load and
maximal load would be useful for comparison purposes.

On Fri, Mar 5, 2021 at 8:09 AM Tom Hill  wrote:

> On 05/03/2021 00:26, Eric Kuhnke wrote:
> > A great deal of this discussion could be resolved by the use of a $20
> > in-line 120VAC watt meter [1] plugged into something as simple as a $500
> > 1U server with some of the DPDK-enabled network cards connected to its
> > PCI-E bus, running DANOS.
>
> I'm fairly sure Etienne-Victor's email made specific reference to
> wattage measurements in both [2] and [3]. It would be fair to assume
> that the authors of those (IEEE) papers understood that you could
> measure wattage at the wall socket, before embarking on a paper
> regarding power efficiency.
>
> > Characterizing the idle load, average usage load, and absolute maximum
> > wattage load of an x86-64 platform is excessively difficult or
> complicated.
>
> It really isn't, particularly when the high figure is 400% of the low
> figure. You don't need milliwatt precision to see that your CPU is
> wasting power while not actually forwarding any packets.
>
> --
> Tom
>

Re: DPDK and energy efficiency

[Eric Kuhnke]

For comparison purposes, I'm curious about the difference in wattage
results between:

a) Your R640 at 420W running DPDK

b) The same R640 hardware temporarily booted from a Ubuntu server live USB,
in which some common CPU stress and memory disk/IO benchmarks are being run
to intentionally load the system to 100% to characterize its absolute
maximum AC load wattage.

https://packages.debian.org/search?keywords=stress

https://packages.debian.org/search?keywords=stress-ng

What's the delta between the 420W and absolute maximum load the server is
capable of pulling on the 208VAC side?

https://manpages.ubuntu.com/manpages/artful/man1/stress-ng.1.html


One possible factor is whether ESXI is configured to pass the pci-e devices
directly through to the guest VM, or if there is any abstraction in
between. For non-ESXI stuff, in the world of Xen or KVM there's many
different ways that a guest domU can access a dom0's network devices, some
of which can have impact on overall steady-state wattage consumed by the
system.

If the greatest possible efficiency is desired for a number of 1U things,
one thing to look at would be something similar to the open compute
platform single centralized AC to DC power units, and servers that don't
each have their own discrete 110-240VAC single or dual power supplies. In
terms of cubic meters of air moved per hour vs wattage, the fans found in
1U servers are really quite inefficient. As a randomly chosen example of
12VDC 40mm (1U server height) fan:

https://www.shoppui.com/documents/9HV0412P3K001.pdf

If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's a
possible load of up to 18.24W for just *one* 40mm height fan. And your
typical high speed dual socket 1U server may have up to eight or ten of
those, in the typical front to back wind tunnel configuration. Normally
fans won't be running at full speed, so each one won't be a 18W load, but
more like 10-12W per fan is totally normal. Plus two at least two more fans
in both hot swap power supplies. Under heavy load I would not be surprised
at all to say that 80W to 90W of your R640's total 420W load is
ventilation.

In a situation where you're running out of power before you run out of rack
space, look at some 1.5U and 2U high chassist that use 60mm height fans,
which are much more efficient in ratio of air moved per time period vs
watts.





On Fri, Mar 5, 2021 at 12:44 PM Brian Knight via NANOG 
wrote:

> On 2021-03-05 12:22, Etienne-Victor Depasquale wrote:
>
> > Sure, here goes:
> >
> > https://www.surveymonkey.com/results/SM-BJ9FCT6K9/
>
> Thanks for sharing these results.  We run DPDK workloads (Cisco nee
> Viptela vEdge Cloud) on ESXI.  Fwiw, a quick survey of a few of our Dell
> R640s running mostly vEdge workloads shows the PS output wattage is
> about 60% higher than a non-vEdge workload: 420W vs 260W.  PS input
> amperage is 2.0A@208V vs 1.4A, a 42% difference.  Processor type is Xeon
> 6152.  Stats obtained from the iDRAC lights-out management module.
>
> vEdge does not do any limiting of polling by default, and afaik the
> software has no support for any kind of limiting.  It will poll the
> network driver on every core assigned to the VM for max performance,
> except for one core which is assigned to the control plane.
>
> I'm usually more concerned about the lack of available CPU cores.  The
> CPU usage forces us not to oversubscribe the VM hosts, which means we
> must provision vEdges less densely and buy more gear sooner.  Plus, the
> increased power demand means we can fit about 12 vEdge servers per
> cabinet instead of 17.  (Power service is 30A 208V, maximum of 80%
> usage.)
>
> OTOH, I face far fewer questions about vEdge Cloud performance problems
> than I do on other virtual platforms.
>
>
> > Cheers,
> >
> > Etienne
>
>
> Thanks again,
>
> -Brian
>

Microsoft Exchange zero day

[Eric Kuhnke]

ISPs/NSPs with customers running self hosted or on-premises Exchange may
want to be aware of this.


https://krebsonsecurity.com/2021/03/at-least-3-u-s-organizations-newly-hacked-via-holes-in-microsofts-email-software/

https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/

Re: DPDK and energy efficiency

[Eric Kuhnke]

> Server PS maximum input wattage is 900W.  Present draw of 2.0A @ 208V is
~420W, so 420/900 = 46.67%

But in the real world an R640 would *never* draw 900W. Even if you were to
load it up with the maximal CPU configuration (2 x 125W TDP CPU per
socket), a full load of 2.5" 15K spinning drives, maximum RAM, and three
high wattage low-profile PCI-E cards, while simultaneously running CPU, RAM
and disk stress tests, you might get in the neighborhood of 550-600W under
load.

Much the same way that a desktop PC equipped with a nominally "850W" rated
active PFC 80+ gold power supply might be powering a motherboard and CPU
combo with a high CPU TDP, but total power consumption under stress
tests/benchmarks would be nowhere near 850W. That rating exists to ensure
that the power supply isn't running anywhere near its max capacity...



On Fri, Mar 5, 2021 at 3:33 PM Brian Knight  wrote:

> On 2021-03-05 15:40, Eric Kuhnke wrote:
>
> > For comparison purposes, I'm curious about the difference in wattage
> > results between:
> >
> > a) Your R640 at 420W running DPDK
> >
> > b) The same R640 hardware temporarily booted from a Ubuntu server live
> > USB, in which some common CPU stress and memory disk/IO benchmarks are
> > being run to intentionally load the system to 100% to characterize its
> > absolute maximum AC load wattage.
>
> We've got a few more hosts waiting to be deployed that are configured
> almost identically.  I'll see what we can do.
>
> I'm guessing those tests would pull slightly more power than the vEdge
> hosts, just because there's not much disk IO that happens on a
> networking VM.  These hosts have four SSDs for local storage.
>
> > What's the delta between the 420W and absolute maximum load the server
> > is capable of pulling on the 208VAC side?
> >
> > https://manpages.ubuntu.com/manpages/artful/man1/stress-ng.1.html
>
> Server PS maximum input wattage is 900W.  Present draw of 2.0A @ 208V is
> ~420W, so 420/900 = 46.67%
>
> > One possible factor is whether ESXI is configured to pass the pci-e
> > devices directly through to the guest VM, or if there is any
> > abstraction in between. For non-ESXI stuff, in the world of Xen or KVM
> > there's many different ways that a guest domU can access a dom0's
> > network devices, some of which can have impact on overall steady-state
> > wattage consumed by the system.
>
> The 420W server has its interfaces routed through the ESXI kernel.
> We're moving quickly to SR-IOV on new servers.
>
> > If the greatest possible efficiency is desired for a number of 1U
> > things, one thing to look at would be something similar to the open
> > compute platform single centralized AC to DC power units, and servers
> > that don't each have their own discrete 110-240VAC single or dual power
> > supplies. In terms of cubic meters of air moved per hour vs wattage,
> > the fans found in 1U servers are really quite inefficient. As a
> > randomly chosen example of 12VDC 40mm (1U server height) fan:
> >
> > https://www.shoppui.com/documents/9HV0412P3K001.pdf
> >
> > If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's
> > a possible load of up to 18.24W for just *one* 40mm height fan. And
> > your typical high speed dual socket 1U server may have up to eight or
> > ten of those, in the typical front to back wind tunnel configuration.
> > Normally fans won't be running at full speed, so each one won't be a
> > 18W load, but more like 10-12W per fan is totally normal. Plus two at
> > least two more fans in both hot swap power supplies. Under heavy load I
> > would not be surprised at all to say that 80W to 90W of your R640's
> > total 420W load is ventilation.
>
> Which is of course dependent on the environmentals.  Fan speeds on our
> two servers are 25% for the 260W vs. 29% for 420W, so not much
> difference.  Inlet temp on both is ~17C.
>
> I checked out another R640 heavily loaded with vEdge VMs, and it's
> pulling similar power, 415W, but the fan speed is at 45%, because inlet
> temp is 22C.
>
> The TDP for the Xeon 6152 is 140W, which seems middle-of-the-road.  From
> the quick survey I did of Dell's configurator, the R640 can take CPUs up
> to 205W.  So we have headroom in terms of cooling.
>
> > In a situation where you're running out of power before you run out of
> > rack space, look at some 1.5U and 2U high chassist that use 60mm height
> > fans, which are much more efficient in ratio of air moved per time
> > period vs watts.
>
> Or ask the colo to turn the A/C lower ;)  (that moves the power problem
> elsewhere, I know)
>
> Thanks,
>
> -Brian
>

RE: OVH datacenter SBG2 in Strasbourg on fire ????

[eric-list]

> -Original Message-
> From: NANOG  On Behalf Of 
> David Hubbard
> Sent: Friday, March 12, 2021 9:47 AM
> To: nanog@nanog.org
> Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 
>
> After sending them abuse reports for years with only an increase in malicious 
> traffic, I have no expectation of anything they do getting better or being 
> for the benefit of the internet as a > whole.  Only reason this is probably 
> getting any attention from them is in hopes they don’t irreparably damage 
> their IPO; they seem to have no issues with their customers' compromised 
> servers damaging the businesses of others on a continuous basis.  

Based on previous outages, I wouldn't agree. 
http://status.ovh.net/?do=details&id=15162#comment18119
I will agree about the spam issues, but Octave Klaba has usually been pretty 
honest comparatively with what I'm used to from typical US based ILECs on 
outages.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

Re: an IP hijacking attempt

[Eric Kuhnke]

I would encourage anyone who is not familiar with the full situation to
read the recent history of AFRINIC events:

https://afrinic.net/ast/pdf/afrinic-whois-audit-report-full-20210121.pdf

https://afrinic.net/20200826-ceo-statement-on-ip-address-misappropriation

https://krebsonsecurity.com/2019/12/the-great-50m-african-ip-address-heist/

On Wed, Mar 17, 2021 at 2:09 AM Brian Turnbow via NANOG 
wrote:

> Hi Noah,
>
>
> > Would you care to share the said prefix?
>
> This is the prefix we found associated with their name in the afrinic db.
>
> inetnum:169.239.204.0 - 169.239.207.255
>
>
> Cheers,
>
> Brian
>
>

Re: Perhaps it's time to think about enhancements to the NANOG list...?

[Eric Kuhnke]

Perhaps the sales, marketing and 'business development' people who've never
typed "enable" or "configure" into a router a single day in their lives
might be better served with a dedicated list that is mission focused on
bizdev, and not operational issues.



On Thu, Mar 18, 2021 at 3:29 PM Matthew Petach 
wrote:

>
> On Thu, Mar 18, 2021 at 10:37 AM Tom Beecher  wrote:
>
>> CC back to the mailing list for visibility, since I ate the CC list.
>>
>> On Thu, Mar 18, 2021 at 1:31 PM Tom Beecher  wrote:
>>
>>> Rod-
>>>
>>> Please refer to the usage guidelines found here.
>>> https://nanog.org/resources/usage-guidelines/
>>>
>>> 14. Posts that encourage or facilitate an agreement about the following
 subjects are inappropriate: prices, discounts, or terms or conditions of
 sale; salaries; benefits, profits, profit margins, or cost data; market
 shares, sales territories, or markets; allocation of customers or
 territories; or selection, rejection, or termination of customers or
 suppliers.
>>>
>>>
>>>  I would tend to agree that while most of your posts to the list are
>>> within the guidelines, there have been occasions where a reasonable person
>>> could think you might be skirting the line a bit. In this case :
>>>
>>> - Your company works as a broker to procure capacity for others.
>>> - You sent an email to the list that wording wise would be exactly the
>>> same as many of us might send to someone they were looking for capacity
>>> from.
>>>
>>> I think most would agree this is pretty clearly against both the usage
>>> guidelines and the spirit of what this mailing list is about.
>>>
>>> I would also like to remind you that this list is administered by the
>>> NANOG organization. You have no authority to tell others to 'cease and
>>> desist', and insult someone as 'underemployed' is also not well tolerated
>>> here.
>>>
>>> I have looped in the list admins here. It would probably be a good idea
>>> to refrain from future messages that are clearly commercial in nature, or
>>> that contain unnecessary insults.
>>>
>>
>
>
> If only we had some way to segregate out different topics
> of interest or disinterest, so that people who weren't interested
> in questions about bandwidth availability could unsubscribe
> from those topics, and only subscribe to the topics that *did*
> interest them...
>
> #AFewDaysTooEarly
>
> ^_^;;
>
> Matt
>
>
>

Re: Perhaps it's time to think about enhancements to the NANOG list...?

[Eric Kuhnke]

In my opinion we have two very different types of 'contact me off list'
things going on here.

We have commercial solicitations and people looking to make contacts for
buying transport circuits, capacity, etc.

And then on the other hand we have 'contact me off list' asks related to
network operational issues, when the subject pertains to what's going on
inside a particular AS, or in some peering or traffic routing problem. Not
everybody wants their own or their peer's dirty laundry aired in public if
something is broken or misconfigured in their relationship to the rest of
the global routing table. And particularly not in a context where it will
go in a public archive forever.




On Sat, Mar 20, 2021 at 4:53 PM Brielle  wrote:

>
> > On Mar 20, 2021, at 3:44 PM, Tom Beecher  wrote:
> >
> > A large portion of these emails lately have contained some variation
> > of 'contact me off list'. How does that provide any benefit to the
> > community? Is anyone else in the community getting any information
> > about what providers may be on a pathway that would help them? They
> > are not.
>
> This is how I’m viewing a lot of this too.  It’s like the posts on stack
> exchange et al, Reddit, and various forums that are just closed with
> “fixed” and no details or follow up.
>
> Kinda defeats the whole purpose of a mailing list with an archive since
> the same questions can come up again and again and no actual answers to the
> questions.
>
>
>
>
>

Re: Perhaps it's time to think about enhancements to the NANOG list...?

[Eric Kuhnke]

It's one thing to use a GUI tool when it's convenient and quick. I think
anyone that's ever experienced setting up a Unifi controller would probably
prefer provisioning a new 802.11ac AP from the GUI rather than doing it
manually at a command line.

But it's another thing to consider that we have a whole new generation of
people who *don't know and don't care* what's going underneath the GUI and
might not be able to do anything with the OS running on bare metal, if they
have to.

If we intend to abstract away configuring devices to a GUI level only and
not care about what's going on under the hood, then it's time for everyone
to just run out and renew their MCSE certifications and buy Meraki
subscriptions.



On Sat, Mar 20, 2021 at 6:35 PM David Siegel  wrote:

>
> ...not to mention that all mature networks are moving more towards GUI
> front ends for their automated network.  As the complexity of a network
> increases, CLI access becomes considerably more risky.
>
> The idea that "real engineers use the CLI" is dinosaur thinking that will
> eventually land those with that philosophy out of a job.  Just my personal
> $.02 (though I'm certainly not alone in my opinion).
>
> But I'd like to reiterate that the board's goal with modernization is not
> to alienate anyone from the existing community by forcing them into a
> web-interface.  Discourse is under evaluation, and if it doesn't accomplish
> the goal we'll try something else or build our own tool.
>
> Dave
>
>
>
>
> On Sat, Mar 20, 2021 at 6:52 PM Matthew Petach 
> wrote:
>
>>
>>
>> On Sat, Mar 20, 2021 at 5:13 PM scott  wrote:
>> [...]
>>
>>>  Of course, one would
>>> not find an HTTP GUI on the bigger networks dealt with on this list;
>>> only on the tiny networks.  So they're beginning learners and are, of
>>> course, welcome.  They will lean a lot, just as I did in the early days
>>> and do every day now days.
>>
>> [...]
>>
>>> scott
>>>
>>
>> Let's see...
>> Google: Gmail
>> Microsoft: Hotmail/Outlook/Office365
>> Yahoo/VerizonMedia: Yahoo Mail
>>
>> I'd have to say, there's some pretty big networks on this list that
>> use HTTP GUIs for their email.
>>
>> Of course, you might be big enough that you look down on the
>> networks of Google, Microsoft, and VZM as "tiny networks" -- in
>> which case, you're definitely entitled to your opinion, as all 8000
>> pound gorillas that look down on the puny 800 lb gorillas are.  ;)
>>
>> Matt
>>
>>
>

Re: OT: Re: Younger generations preferring social media(esque) interactions.

[Eric Kuhnke]

For persons considering mattermost, I would recommend instead looking into
a self hosted Matrix + Synapse (matrix protocol server daemon) setup, which
is fully open source.

https://en.wikipedia.org/wiki/Matrix_(protocol)

Element is one typical GUI client for it, but there are many options.
https://en.wikipedia.org/wiki/Element_(software)



On Mon, Mar 22, 2021 at 11:45 PM Cynthia Revström via NANOG 
wrote:

> I have used Mattermost but iirc it has very limited access control unless
> you have the enterprise version and generally doesn't seem to be made for
> public groups.
>
> This in addition to probably the main problem, it will have higher barrier
> to entry especially for those already using Discord for other purposes.
>
> -Cynthia
>
> On Tue, Mar 23, 2021, 07:38 Karl Auer  wrote:
>
>> On Tue, 2021-03-23 at 07:22 +0100, Cynthia Revström via NANOG wrote:
>> > Because Discord is proprietary, you can't host your own instance
>>
>> For reasons of confidentiality we implemented a MatterMost server for
>> company use. It is free, works well, runs on our own servers. It lacks
>> some of the bells and whistles that Discord has (in particular it has
>> no audio or screensharing), but as an instant messaging platform for us
>> it's worked very well indeed.
>>
>>
>> Regards, K.
>>
>> --
>> ~~~
>> Karl Auer (ka...@biplane.com.au)
>> http://www.biplane.com.au/kauer
>>
>> GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
>> Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D
>>
>>
>>
>>

Re: IP reputation lookup (prefix not single IP)

[Eric Kuhnke]

I think you will find that most SMTP / anti-spam focused RBL tools give a
very similar result for IP reputation on a per /24 block basis, for any
randomly chosen IP in the block, particularly where the /24 in question has
previously been used and announced by a dedicated server/VPS/virtual server
hosting company.


On Thu, Mar 25, 2021 at 9:26 AM vom513  wrote:

> Hello all,
>
> I’ve seen other folks asking the same/similar question in the past, but I
> don’t recall seeing more than a few options out there to *try* to suss this
> out.  Use case is someone I’m working with looking to buy a v4 block from a
> broker.
>
> So far I’ve checked Talos and Sorbs (both allow a prefix lookup).  Most of
> the other RBL/multi-RBL sites want a single IP (the use case being email of
> course).  I won’t abuse their service by trying to lookup each single IP in
> the block...
>
> Could anyone share anything/anywhere else I might look to get crumbs of
> info on a given preifx ?
>
> Thanks.

Re: IP reputation lookup (prefix not single IP)

[Eric Kuhnke]

Nothing more than anecdotal evidence, when I last looked into the
externally available network details on a number of low-budget VPS hosting
companies...   I would say that if anything, a person who really knows what
they're doing operating a properly MX, will face more difficulties today
than they did 3, 5 or 7 years ago operating the system in the same
netblocks as IPs which have been previously abused.

For obvious reasons the IP reputation systems and antispam tools at the
biggest destinations (gsuite/gmail, office365, etc) are treated as closely
guarded proprietary data.

My personal theory on a whole /24 acquiring a poor reputation, is that it
does have some correlation with the density of random $5/mo VPS customers
and the turnover of different customers between the same small group of
IPs. And exactly how many misconfigured smtp sources have existed in that
block within some previous range of time, how much spam has been
reported/flagged, etc.



On Thu, Mar 25, 2021 at 8:28 PM Randy Bush  wrote:

> > I think you will find that most SMTP / anti-spam focused RBL tools
> > give a very similar result for IP reputation on a per /24 block basis
>
> got cites?  this got me curious the other day.
>
> randy
>
> ---
> ra...@psg.com
> `gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
> signatures are back, thanks to dmarc header butchery
>

Re: 10 years from now... (was: internet futures)

[Eric Kuhnke]

I would also concur that the likelihood of Starlink (or a Oneweb, or
Kuiper) terminal being used successfully to bypass the GFW or similar
serious Internet censorship, in an authoritarian environment, is probably
low. This is because:

a) It has to transmit in known bands.

b) It has to be located in a location with a very good, clear view of the
sky in all directions (even a single tree obstruction in one section of the
sky, relative to where the antenna is mounted will cause packet
loss/periodic issues on a starlink beta terminal right now). Visually
identifying the terminal would not be hard.

c) Portable spectrum analyzers capable of up to 30 GHz are not nearly as
expensive as they used to be. They also have much better GUIs and
visualization tools than what was available 6-10 years ago.

d) You could successfully train local law enforcement to use these sort of
portable spectrum analyzers in a one-day, 8-hour training course.

e) The equipment would have to be smuggled into the country

f) Many people such as in a location like Iran may lack access to a
standard payment system for the services (the percentage of Iranians with
access to buy things online with visa/mastercard/american express or
similar is quite low).


There are already plenty of places in the world where if you set up a 1.2,
1.8 or 2.4 meter C, Ku or Ka band VSAT terminal using some sort of
geostationary based services, without appropriate government "licenses",
men with guns will come to dismantle it and arrest you.

I am not saying it is an impossible problem to solve, but any system
intended for that sort of purpose would have to be designed for
circumvention, and not a consumer/COTS adaptation of an off the shelf
starlink terminal.









On Sat, Mar 27, 2021 at 8:31 PM na...@jima.us  wrote:

> Please don't forget that RF sources can be tracked down by even
> minimally-well-equipped adversaries.
>
> - Jima
>
> -Original Message-
> From: NANOG  On Behalf Of scott
> Sent: Saturday, March 27, 2021 19:36
> To: nanog@nanog.org
> Subject: Re: 10 years from now... (was: internet futures)
>
>
> On 3/26/2021 9:42 AM, Michael Thomas wrote:
> > LEO internet providers will be coming online which might make a
> > difference in the corners of the world where it's hard to get access,
> > but will it allow internet access to parachute in behind the Great
> > Firewall?
> 
> > How do the Chinas of the world intend to deal with the Great Firewall
> > implications?
>
>
> This is what I hope will change in the next 10 years.  "Turning off the
> internet" will be harder and harder for folks suppressing others, many
> times violently, and hiding it from everyone else.  A small-ish antenna
> easily hidden would be necessary.
>
> scott
>
>
>
>

Re: 10 years from now... (was: internet futures)

[Eric Kuhnke]

By definition and orbital mechanics, low earth orbit things don't "park"
anywhere. There's an equal number of starlink satellites over Mongolia
right now as there are over the same latitude locations in the US and
Canada.

https://satellitemap.space/

This also becomes intuitive once one plays Kerbal Space Program for a few
hours...




On Sun, Mar 28, 2021 at 9:16 PM Keith Medcalf  wrote:

>
> Net to mention, of course, that the Low Orbit constellation would need to
> be "parked" over China (or where-ever you want to access it).  I am quite
> sure that "shooting down" such low orbit stationary vehicles would not be
> too difficult.  And if they are owned by an adversary who has no permission
> to fly those objects in your airspace, I doubt that anything could be done
> about it.
>
> If I owned a bunch of low orbit satellites costing millions of dollars
> each, I would not want to "park" them in low orbit over a hostile territory.
>
> Then you also have the requirement to maintain positive control over the
> satellites which, unlike those in geostationary orbits, need to be under
> continual thrust and control in order to stay "parked".  I doubt that any
> "private" (ie, non-Government organization) could afford to do so without
> the cooperation of the state over which they are parking.
>
> --
> Be decisive.  Make a decision, right or wrong.  The road of life is paved
> with flat squirrels who could not make a decision.
>
> >-Original Message-
> >From: NANOG  On Behalf Of
> >Eric Kuhnke
> >Sent: Sunday, 28 March, 2021 18:24
> >To: na...@jima.us
> >Cc: nanog@nanog.org
> >Subject: Re: 10 years from now... (was: internet futures)
> >
> >I would also concur that the likelihood of Starlink (or a Oneweb, or
> >Kuiper) terminal being used successfully to bypass the GFW or similar
> >serious Internet censorship, in an authoritarian environment, is probably
> >low. This is because:
> >
> >a) It has to transmit in known bands.
> >
> >
> >b) It has to be located in a location with a very good, clear view of the
> >sky in all directions (even a single tree obstruction in one section of
> >the sky, relative to where the antenna is mounted will cause packet
> >loss/periodic issues on a starlink beta terminal right now). Visually
> >identifying the terminal would not be hard.
> >
> >
> >c) Portable spectrum analyzers capable of up to 30 GHz are not nearly as
> >expensive as they used to be. They also have much better GUIs and
> >visualization tools than what was available 6-10 years ago.
> >
> >
> >d) You could successfully train local law enforcement to use these sort
> >of portable spectrum analyzers in a one-day, 8-hour training course.
> >
> >
> >e) The equipment would have to be smuggled into the country
> >
> >f) Many people such as in a location like Iran may lack access to a
> >standard payment system for the services (the percentage of Iranians with
> >access to buy things online with visa/mastercard/american express or
> >similar is quite low).
> >
> >
> >
> >There are already plenty of places in the world where if you set up a
> >1.2, 1.8 or 2.4 meter C, Ku or Ka band VSAT terminal using some sort of
> >geostationary based services, without appropriate government "licenses",
> >men with guns will come to dismantle it and arrest you.
> >
> >I am not saying it is an impossible problem to solve, but any system
> >intended for that sort of purpose would have to be designed for
> >circumvention, and not a consumer/COTS adaptation of an off the shelf
> >starlink terminal.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On Sat, Mar 27, 2021 at 8:31 PM na...@jima.us <mailto:na...@jima.us>
> >mailto:na...@jima.us> > wrote:
> >
> >
> >   Please don't forget that RF sources can be tracked down by even
> >minimally-well-equipped adversaries.
> >
> >   - Jima
> >
> >   -Original Message-
> >   From: NANOG  ><mailto:jima...@nanog.org> > On Behalf Of scott
> >   Sent: Saturday, March 27, 2021 19:36
> >   To: nanog@nanog.org <mailto:nanog@nanog.org>
> >   Subject: Re: 10 years from now... (was: internet futures)
> >
> >
> >   On 3/26/2021 9:42 AM, Michael Thomas wrote:
> >   > LEO internet providers will be coming online which might make a
> >   > difference in the corners of the world where it's hard to get
> >access,
> >   > but will it allow internet access to parachute in behind the
> Great
> >   > Firewall?
> >   
> >   > How do the Chinas of the world intend to deal with the Great
> >Firewall
> >   > implications?
> >
> >
> >   This is what I hope will change in the next 10 years.  "Turning off
> >the
> >   internet" will be harder and harder for folks suppressing others,
> >many
> >   times violently, and hiding it from everyone else.  A small-ish
> >antenna
> >   easily hidden would be necessary.
> >
> >   scott
> >
> >
> >
> >
>
>
>
>
>

Re: 10 years from now... (was: internet futures)

[Eric Kuhnke]

The US State Department is already a large customer for dedicated
transponder capacity, in C-band hemispheric and Ku beams in some weird
places in the world.

As a randomly chosen example if you take a look at the roof of the UK
embassy in Kabul, there's a nice 4 meter size Andrew/Commscope compact
cassegrain dish up there. Pretty typical thing already for embassies, the
big difference would be that that they'll have more market options for
high-throughput service.


On Sun, Mar 28, 2021 at 10:18 PM Mark Tinka  wrote:

>
>
> On 3/29/21 02:23, Eric Kuhnke wrote:
>
> >
> > I am not saying it is an impossible problem to solve, but any system
> > intended for that sort of purpose would have to be designed for
> > circumvention, and not a consumer/COTS adaptation of an off the shelf
> > starlink terminal.
>
> Behind the walls of an embassy, perhaps :-).
>
> Mark.
>

Re: 10 years from now...

[Eric Kuhnke]

The present architecture is logically a bent pipe, where a moving satellite
(preferably more than one, for make before break handoff function) needs to
be simultaneously in view of a starlink earth station and the CPE.

In the long term this may not be an absolute. Ten beta test satellites that
were launched into a near polar orbit a few months back have test equipment
on them for inter-satellite laser links.

Satellite to Satellite relay by Ka-band for low bandwidth stuff has been
demonstrated and in production for a long time. For quite a while the only
two Iridium earth stations existed in Arizona and Hawaii. A handheld phone
call or SMS from an Iridium terminal anywhere in the world would make its
way through the satellite network to those locations. Statements by Musk
indicate that they have a strong desire for a long term ability to do
something like that, but optically and with much higher throughput. I would
also be surprised if Kuiper does not have similar intentions.



On Sun, Mar 28, 2021 at 9:05 PM  wrote:

> This is a fascinating discussion.
>
> Also keep in mind that starlink satellites need many earth stations to
> downlink customer packets and provide internet transit. There are over
> 50 satellite earth stations in the US already.
>
> Here is a great google map of the current ground stations based on FCC
> license data:
>
> https://www.google.com/maps/@36.6554578,-111.9151229,4.5z/data=!4m2!6m1!1s1H1x8jZs8vfjy60TvKgpbYs_grargieVw
>
> -Keith
>
> Denys Fedoryshchenko wrote on 3/28/2021 6:58 PM:
>
> > No need for all that fancy RF tools.
> > Moreover, detecting >10Ghz transmission is not such an easy task.
> > The beam is most likely narrow enough to be difficult to detect.
> >
> > But, (for example) it's enough to visit from foreign IPs some local
> > website,
> > to have cookie set: SATELLITE_USER=xyz
> > Then when person use local connection and visit same website, this cookie
> > will send law enforcement hint.
> > And there are many more automated, software-based ways to detect that a
> > device has been connected via satellite in past.
> >
> > Not to mention the fact that any attempt to provide services illegally
> > is pandora box.
> > At least it may end up with the fact that the country will start
> > jamming uplink
> > frequencies, which will affect the service in whole region.
> > And in the worst case, it will give reason to use anti-satellite weapons.
> >
> >
> > On 2021-03-29 03:23, Eric Kuhnke wrote:
> >> I would also concur that the likelihood of Starlink (or a Oneweb, or
> >> Kuiper) terminal being used successfully to bypass the GFW or similar
> >> serious Internet censorship, in an authoritarian environment, is
> >> probably low. This is because:
> >>
> >> a) It has to transmit in known bands.
> >>
> >> b) It has to be located in a location with a very good, clear view of
> >> the sky in all directions (even a single tree obstruction in one
> >> section of the sky, relative to where the antenna is mounted will
> >> cause packet loss/periodic issues on a starlink beta terminal right
> >> now). Visually identifying the terminal would not be hard.
> >>
> >> c) Portable spectrum analyzers capable of up to 30 GHz are not nearly
> >> as expensive as they used to be. They also have much better GUIs and
> >> visualization tools than what was available 6-10 years ago.
> >>
> >> d) You could successfully train local law enforcement to use these
> >> sort of portable spectrum analyzers in a one-day, 8-hour training
> >> course.
> >>
> >> e) The equipment would have to be smuggled into the country
> >>
> >> f) Many people such as in a location like Iran may lack access to a
> >> standard payment system for the services (the percentage of Iranians
> >> with access to buy things online with visa/mastercard/american express
> >> or similar is quite low).
> >>
> >> There are already plenty of places in the world where if you set up a
> >> 1.2, 1.8 or 2.4 meter C, Ku or Ka band VSAT terminal using some sort
> >> of geostationary based services, without appropriate government
> >> "licenses", men with guns will come to dismantle it and arrest you.
> >>
> >> I am not saying it is an impossible problem to solve, but any system
> >> intended for that sort of purpose would have to be designed for
> >> circumvention, and not a consumer/COTS adaptation of an off the shelf
> >> starlink terminal.
> >>
> >> On Sat, Mar 27, 2021 at 8:31 PM na...@jima.us  wrote:
>

Re: 10 years from now... (was: internet futures)

[Eric Kuhnke]

I am doing this right now. A starlink CPE is a fairly ordinary DIA link
that exists in cgnat space from the perspective of whatever router you plug
into it. The starlink indoor 'router' is optional.

Whatever you plug into the high power PoE injector will be given a DHCP
lease and a default route out to the world. People who don't want to DIY a
dual-WAN solution with their own Linux box or pfsense or similar can use
things like peplink routers.

In difficult to reach places in the world I can see starlink with a
low-bandwidth traditional VSAT link as backup/failover as a fairly common
configuration. Or starlink as primary and local HSPA+/HSDPA/LTE (whatever
3GPP related) as a failover.



On Mon, Mar 29, 2021 at 12:37 PM Matt Erculiani 
wrote:

> I wouldn't be the least bit surprised if anyone out there was trying to
> mix their StarLink kit and existing broadband service to optimize
> performance and/or add redundancy though.
>
> The underlying technologies will change, but what people try to do with
> them will remain relatively unchanged.
>
> Back 20 years ago people were talking about their Frame Relay P2P
> services, now they talk about their Ethernet P2P services.
>
> -Matt
>
> On Mon, Mar 29, 2021 at 1:10 PM Aaron C. de Bruyn 
> wrote:
>
>> On Mon, Mar 29, 2021 at 11:39 AM Matt Erculiani 
>> wrote:
>>
>>> I think the best way to think about what 10 years from now will look
>>> like is to compare 10 years ago to the present:
>>> https://mailman.nanog.org/pipermail/nanog/2011-April/thread.html
>>>
>>
>> Multi-homing your DSL connection?
>> I can't wait to multi-home my 10x10 array of StarLink satellites in a few
>> years...
>>
>> -A
>>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>

Re: My First BGP-Hijacking Explanation

[Eric Kuhnke]

If one follows the social media accounts of the Pakistan version of the
FCC, nowadays they're just banning anything they find insulting or illegal
in the local legal system, and ordering ISPs to null route big chunks of IP
space.

As an anecdotal data point, the only effect this has had is teaching random
14 year olds how to use ordinary consumer grade VPNs, which work just fine.

https://www.pta.gov.pk/en



On Thu, Apr 8, 2021 at 9:52 AM Jay R. Ashworth  wrote:

> Sam 'Half As Interesting' Denby actually did a surprisingly good job
> explaining
> this for the average only-vaguely-technical viewer...
>
>https://www.youtube.com/watch?v=K9gnRs33NOk
>
> [ For all the bad dad jokes he tells on HAI, he's got really good research
>   skills/staff, and his long-form stuff on Wendover Productions is
> excellent ]
>
>
> Cheers,
> -- jra
>
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land
> Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>

Re: OOB management options @ 60 Hudson & 1 Summer

[Eric Kuhnke]

Before getting rid of the cellular based OOB, look into some more detail
about exactly what LTE modems are in those. I've seen some remarkable
results from equipment using the 600/700 bands (tmobile, verizon) for
getting signal into deeply buried concrete structures. There's a lot of
different types and capabilities of cellular data modems on the market.




On Thu, Apr 15, 2021 at 3:15 PM Matthew Crocker 
wrote:

>
>
> I have routers in both 60 Hudson St & 1 Summer St and I’m looking for some
> low cost bandwidth options for out of band management.  Currently I have
> Opengear boxes at each site with cell modems but they don’t work too well.
> I either need to replace them with new cell based devices or find a
> wireless/ethernet bandwidth option.   I only need a couple serial ports and
> ethernet for when everything breaks.
>
>
>
> I’m in DR space @ 60 Hudson and the Markeley MMR @ 1 Summer
>
>
>
> I’m surprised OOB bandwidth isn’t a feature for colocation providers.
>
>
>
> Thanks
>
>
>

Malicious SS7 activity and why SMS should never by used for 2FA

[Eric Kuhnke]

https://lucky225.medium.com/its-time-to-stop-using-sms-for-anything-203c41361c80

https://krebsonsecurity.com/2021/03/can-we-stop-pretending-sms-is-secure-now/


Anecdotal: With the prior consent of the DID holders, I have successfully
ported peoples' numbers using nothing more than a JPG scan of a signature
that looks like an illegible 150 dpi black and white blob, pasted in an
image editor on top of a generic looking 'phone bill'.

Re: Malicious SS7 activity and why SMS should never by used for 2FA

[Eric Kuhnke]

One of my main problems with SMS 2FA from a usability standpoint, aside
from SS7 hijacks and security problems, is that it cannot be relied upon
when traveling in many international locations. I have been *so many places*
where there is just about zero chance of my T-Mobile SIM successfully
roaming onto the local network and receiving SMS at my US or Canadian
number successfully.

What am I supposed to do, take the SIM out of my phone, put it in a burner
and give it to a trusted family member in North America, just for the
purpose of receiving SMS 2FA codes (which I then have to call them and get
the code from manually each time), before going somewhere weird?

In the pre covid19 era when people were actually traveling places, imagine
you've had reason to go somewhere weird and need access to a thing (such as
your online banking, perhaps?) protected by SMS 2FA, but you have
absolutely no way of receiving the SMS where you're presently located...

Many of the people designing SMS 2FA systems used by people with
accounts/services in the US 50 states and Canada seem to assume that their
domestic customers will forever remain in a domestic location.




On Sun, Apr 18, 2021 at 5:44 AM Mark Tinka  wrote:

>
>
> On 4/18/21 05:18, Mel Beckman wrote:
>
> > No, every SMS 2FA should be prohibited by regulatory certifications.
> > The telcos had years to secure SMS. They did nothing. The plethora of
> > well-secured commercial 2FA authentication tokens, many of them free,
> > should be a mandatory replacement for 2FA in every security governance
> > regime, such as PCI, financial account access, government web portals,
> > etc.
>
> While I agree that SMS is insecure at the moment, I think there still
> needs to be a mechanism that does not rely on the presence of an
> Internet connection. One may not be able to have access to the Internet
> for a number of reasons (traveling, coverage, outage, device, money,
> e.t.c.), and a fallback needs to be available to authenticate.
>
> I know some companies have been pushing for voice authentication for
> their services through a phone call, in lieu of SMS or DTMF-based PIN's.
>
> We need something that works at the lowest common denominator as well,
> because as available as the Internet is worldwide, it's not yet at a
> level that one would consider "basic access".
>
> Mark.
>

Re: Malicious SS7 activity and why SMS should never by used for 2FA

[Eric Kuhnke]

I would start with cellular carriers and nations that intentionally take
steps to block anything VoIP as a threat to their revenue model. Or because
anything vpn/ipsec/whatever related is a threat to local Internet
censorship laws.

Plenty of places the sort of ipsec tunnel used for vowifi is not usable on
whatever consumer-grade cellular or local broadband ISP you might find.




On Sun, Apr 18, 2021 at 11:11 PM Mark Tinka  wrote:

>
>
> On 4/19/21 06:50, Julien Goodwin wrote:
>
> > This is already probably past the point of being on topic here, but you
> > tickled my personal favorite one of these.
> >
> > My airline of choice (Qantas) has mandatory SMS second factor, after
> > perhaps a mobile carrier requiring it for support one of the most
> > facepalm-worthy uses of SMS 2FA I've seen.
>
> It's interesting that VoWiFi is meant to support both voice and SMS,
> domestically and when one travels. So I'm curious why SMS's would not
> work with VoWiFi when traveling to a country that won't deliver your
> SMS's generically. After all, VoWiFi is, as far as I understand it,
> meant to be a direct IP tunnel back to your home network for both
> billing and service.
>
> If anyone has more clue about this on the list, I'd really like to know,
> as my mobile service providers hardly know what I'm talking about when I
> ring them up with questions.
>
> Mark.
>
>

Re: Submitting Fake Geolocation for blocks to Data Brokers and RIRs

[Eric Kuhnke]

I sincerely doubt that any actual *law* could be enforced against an ISP
which is a legal entity in one location, yet has multiple discrete /23 or
/24 blocks and without any obfuscation choose to announce them from
multiple different geographic locations. Configurations where an AS has
multiple islands of service which are not linked together by an internal
transport network are not that rare these days (see prior discussion about
merits vs risks of filtering out your own netblock at your BGP edge). If
anyone is aware of any case law precedent for such a prosecution it would
be interesting to see citations.

The only scenario in which I could see a legal penalty being imposed on
some ISP, is if it fails to publish an accurate record of its corporate
name, address and contact info for its ARIN, RIPE, AFRINIC, APNIC, etc
entity listing as a corporation. Obviously you can't and shouldn't attempt
to obfuscate where you're headquartered, and you need to be able to prove
your legal entity bona fides to ARIN or RIPE anyways in order to maintain
registration.

As to whether third party content sources might refuse to serve content to
an ISP announcing blocks in weird places, an ISP tunneling a customer's
traffic from one location to another, or misunderstanding their geolocation
(Hulu in the US is a fine example of this, its regional content is broken
on Starlink right now because of a misunderstanding of how the cgnat
traffic meets the real Internet), that's not a law...

That's an arbitrary private choice of some OTT video content provider or
CDN to serve or not serve certain licenses of copyrighted content based on
what it thinks is geolocation data. Another example would be the content
you see on Canadian domestic netflix vs US domestic netflix.



On Wed, Apr 21, 2021 at 12:22 PM William Herrin  wrote:

> On Wed, Apr 21, 2021 at 11:58 AM nanoguser100 via NANOG 
> wrote:
> > I wanted to get the communities' opinion on this.
> >
> > Increasingly I have run into 'niche needs' where a client has a few
> users in a country we don't have a POP, say Estonia.  This is 'mainly' for
> localization but also in some cases for compliance (some sites REQUIRE an
> Estonian IP).  With that being said is it common practice to 'fake'
> Geolocations?  In this case the user legitimately lives in Estonia, they
> just happen to be using our cloud service in Germany.
>
> If the endpoint (e.g. web server) is physically located in Germany and
> you're helping a client misrepresent that it's located in Estonia in
> order to evade a legal requirement that it be located in Estonia then
> you've made yourself a party to criminal fraud. Do I really need to
> explain how bad an idea that is?
>
> If the service is a VPN relay for addresses which are actually being
> used in Estonia then what's the problem? You're just a transit for
> those IPs. Report the location where the endpoints are, not the
> transits.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: Broken Mini-SAS cable removal?

[Eric Litvin]

Joe’s response is spot on. I would also suggest you look at the “latching 
finger” mechanism on a spare,  then apply some of the techniques Joe suggests. 

Eric 
Luma optics 


 

Sent from my iPhone

> On Apr 23, 2021, at 8:27 AM, Joe Klein  wrote:
> 
> Try shim stock or a feeler gauge between the plug and socket to work the 
> latching fingers. This isn't something that I've tried specifically in this 
> case. 
> 
> You might need to put a notch in the stock or feeler gauge so that you can 
> work the fingers from the backside. Kinda like that old trick of using a 
> credit card to prise a door latch, except this should work since there's no 
> deadlatch. :)
> 
> You might also try gently twisting a small screwdriver or spudger stick 
> between the plug and socket too to increase the gap between the socket and 
> plug. 
> 
> -joe
> 
> From: NANOG  On Behalf Of 
> Ryland Kremeier
> Sent: Friday, April 23, 2021 09:31
> To: nanog@nanog.org
> Subject: Broken Mini-SAS cable removal?
> 
>  
>  External Mail
>  
> Anyone here have experience removing a mini-SAS cable when the plastic tab 
> has broken off? Tried checking online but couldn't find anything.
> 
> Thank you,
> -- Ryland
> 

Re: FCC fines for unauthorized carrier changes and consumer billing

[Eric Kuhnke]

Did the FCC ever collect its $50 million from "Sandwich Isles
Telecommunications" for blatant fraud?  At this scale I wonder how or why
certain people are not in federal prison.

https://www.google.com/search?channel=fs&q=fcc+sandwich+isles

https://docs.fcc.gov/public/attachments/FCC-20-131A1_Rcd.pdf

V. ORDERING CLAUSES69. Accordingly,IT IS ORDEREDthat Sandwich Isles
Communications, Inc., Waimana Enterprises, Inc., and Albert S.N. Hee,
pursuant to section 220(d) of the Communications Act of 1934, as amended,
and section 1.80 of the Commission’s rules,293ARE JOINTLY AND SEVERALLY LIABLE
FOR A MONETARY FORFEITUREin the amount of forty-nine million, five hundred
and ninety-eight thousand, and four hundred and forty-eight dollars
($49,598,448) for violating the Act and the Commission’s rules.70. Payment
of the forfeiture shall be made in the manner provided for in section 1.80
of the Commission’s rules within thirty (30) calendar days after the release
of this Forfeiture Order.29

On Fri, Apr 23, 2021 at 8:44 AM Sean Donelan  wrote:

>
> The FCC has a poor record of actually collecting money from Notices of
> Apparent Liability (i.e. fines).  There are flaws in the FCC notification
> rules, but it does have some rules requiring indpendent verification of
> carrier changes.
>
>
>
> FCC Fines Tele Circuit $4,145,000 for Cramming & Slamming Violations
>
>
> https://www.fcc.gov/document/fcc-fines-tele-circuit-4145000-cramming-slamming-violations-0
>
> FCC fines Tele Circuit Network Corporation $4,145,000 for switching
> consumers from their preferred carrier to Tele Circuit without permission
> and adding unauthorized charges to consumers' bill
>
> In this Order, the Federal Communications Commission (FCC or Commission)
> adopts the findings in the Notice of Apparent Liability (Tele Circuit
> Notice or Notice) that Tele Circuit Network Corporation (Tele Circuit or
> Company) engaged in slamming and cramming, made misrepresentations to
> consumers, and violated a Commission order by failing to produce certain
> information and documents relating to the Company’s business practices.
> The Company’s misconduct  harmed elderly and infirm consumers, in some
> cases leaving them without telephone service for extended periods of
> time—with Tele Circuit refusing to reinstate service until the crammed
> charges were paid in full. These practices violated sections 201(b) and
> 258 of the Communications Act of 1934, as amended (the Act), and section
> 64.1120 of the Commission’s rules. After reviewing Tele Circuit’s response
> to the Notice, we decline to find that the Company violated section
> 1.17(a)(2) of the Commission’s rules and reduce the proposed penalty by
> $1,178,322, and therefore impose a forfeiture of $4,145,000.
>
>
>
> [...]
> In particular, Tele Circuit did not provide proof that the complainants
> listed in the LOI authorized Tele Circuit to switch their long distance
> carrier. In response to the LOI, Tele Circuit did produce some
> third-party verification recordings,31 which are supposed to provide
> evidence that customers assented to changing their long distance service
> from their existing carriers to Tele Circuit. However, some
> complainants who listened to the recordings alleged that the third-party
> verifications were falsified. In all, the Bureau reviewed more than 100
> written consumer complaints, contacted numerous complainants, obtained
> substantial documentary evidence (including copies of consumer telephone
> bills), listened to third-partyverification recordings, and received data
> from consumers’ carriers.
>
> [...]
>   Tele Circuit switched the telephone service of 24 consumers without
> verified authorization to do so, in violation of section 258 of the Act
> and section 64.1120 of the Commission’s rules.
>
> [...]
>

Re: Myanmar internet - something to think about if you're having a bad day

[Eric Kuhnke]

It should be noted that Telenor has been one of the nationwide license
holders for 3GPP cellular bands in Pakistan for a long time, and has
encountered the same issues with regional network shutdowns, and government
orders to block certain netblocks or services.

Not to the same extent as what's going on right now in Myanmar, but
absolutely it meets the definition of what a (western European, North
American) person would consider to be unconscionable and unwarranted
government Internet censorship and interference with telecoms.

They've shown no signs of pulling out of Pakistan or making operational
changes as a result of this, over the past ten years. My personal opinion
is that Telenor (PK) has weighed the risks, and judged that they possess
neither the political capital, influence or leverage to ignore the
government's occasional Internet shutdown orders.

"Westerners" might be surprised to learn the extent that some of the major
international/developing-nation specialist 3GPP carriers seem to be quite
fine with operating in non-democratic regimes and bending their telecom's
operational policies to suit local laws. In particular I'm thinking of the
above Telenor example, but also MTN in many nations in Africa, Orange, and
Airtel, in their operations in many different nations.

Then on the other hand you have telecom entities which originate from
highly censored political systems, one of the other 3GPP band operators in
Pakistan (Zong) is owned by a Chinese domestic telecom company.







On Mon, Apr 26, 2021 at 11:51 PM Bjørn Mork  wrote:

> scott  writes:
>
> > Telenor and Ooredoo, it's time to do the right thing.
>
> Wrt Telenor, please see the info posted at
>
> https://www.telenor.com/sustainability/responsible-business/human-rights/mitigate/human-rights-in-myanmar/directives-from-authorities-in-myanmar-february-2021/
>
>
> Bjørn
>

Re: Myanmar internet - something to think about if you're having a bad day

[Eric Kuhnke]

None of them are a good option. In the specific case of Pakistan, the
periodic shutdowns and blockages have been 'moderate' enough, if that's an
appropriate word to use, that *most* of the time, Telenor's customers have
ordinary Internet service. Over the long run it is probably a benefit that
its customers have their LTE data services.

Within that specific example I should also note that there has been very
little effort put on a nation-wide scale to implement technology which can
do DPI and drop/blackholing of VPN traffic. Even though the Internet
traffic for the country runs through a few choke points, there does not
appear to be government-operated technical capability or the budget to
implement something on the scale of the great firewall.

There's plenty of non technical teenagers in Pakistan with VPN clients on
their phone or laptop who seem perfectly capable of using a VPN to watch
Youtube or access Twitter and other social media, during the periods of
time that the government orders things to be blocked.

Along with all feasible attempts at lobbying, I would propose a 4th
alternate to the scenarios outlined, which is to provide funding and
financial support (from a telecom's headquarters in Europe or the USA) for
civil society institutions and non-profits related to bypassing Internet
censorship, and lobbying against it. Such as the EFF, funding for the tor
project, supporting the work of various GPL/BSD licensed VPN technologies
(openvpn, wireguard, etc) and their continuing development, etc.




On Wed, Apr 28, 2021 at 11:03 AM Christopher Morrow 
wrote:

> (I'm sure i'll regret this, but...)
>
> On Wed, Apr 28, 2021 at 1:48 PM Eric Kuhnke  wrote:
>
>> It should be noted that Telenor has been one of the nationwide license
>> holders for 3GPP cellular bands in Pakistan for a long time, and has
>> encountered the same issues with regional network shutdowns, and government
>> orders to block certain netblocks or services.
>>
>> Not to the same extent as what's going on right now in Myanmar, but
>> absolutely it meets the definition of what a (western European, North
>> American) person would consider to be unconscionable and unwarranted
>> government Internet censorship and interference with telecoms.
>>
>>>
>>>
> So, what would be the correct set of actions here (for a company)?
>
> it sounds like some version of the proposal is:
>   "Pull up stakes, stop offering services in places that may/do impose
> 'draconian' methods of 'censorship'"
>  (note intentionally quoted draconian/censorship - I don't mean/want
> to put a value on those words)
>
> or perhaps:
>   "Lobby the gov't(s) in these situations to NOT do the things they keep
> doing"
>
> or finally:
>   "refuse to comply with requests/orders from govt(s) to do these things"
>
> I think the last is 'impractical', I expect the 1st is also a tough pill
> to swallow for a large multinational telcom... the middle may already be
> being done, but is unlikely to help.
>
> So, aside from: you ought not do that! from
> the sidelines... what should a responsible Corpo do?
>

Re: link monitoring

[Eric Kuhnke]

The Junipers on both sides should have discrete SNMP OIDs that respond with
a FEC stress value, or FEC error value. See blue highlighted part here
about FEC. Depending on what version of JunOS you're running the MIB for it
may or may not exist.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB36074&cat=MX2008&actp=LIST

In other equipment sometimes it's found in a sub-tree of SNMP adjacent to
optical DOM values. Once you can acquire and poll that value, set it up as
a custom thing to graph and alert upon certain threshold values in your
choice of NMS.

Additionally signs of a failing optic may show up in some of the optical
DOM MIB items you can poll: https://mibs.observium.org/mib/JUNIPER-DOM-MIB/

It helps if you have some non-misbehaving similar linecards and optics
which can be polled during custom graph/OID configuration, to establish a
baseline 'no problem' value, which if exceeded will trigger whatever
threshold value you set in your monitoring system.

On Thu, Apr 29, 2021 at 1:40 PM Baldur Norddahl 
wrote:

> Hello
>
> We had a 100G link that started to misbehave and caused the customers to
> notice bad packet loss. The optical values are just fine but we had packet
> loss and latency. Interface shows FEC errors on one end and carrier
> transitions on the other end. But otherwise the link would stay up and our
> monitor system completely failed to warn about the failure. Had to find the
> bad link by traceroute (mtr) and observe where packet loss started.
>
> The link was between a Juniper MX204 and Juniper ACX5448. Link length 2
> meters using 2 km single mode SFP modules.
>
> What is the best practice to monitor links to avoid this scenarium? What
> options do we have to do link monitoring? I am investigating BFD but I am
> unsure if that would have helped the situation.
>
> Thanks,
>
> Baldur
>
>
>

Re: link monitoring

[Eric Kuhnke]

If I may add one thing I forgot, this post reminded me. In the question I
think it was probably a 100G CWDM4 short distance link. When monitoring a
100G coherent (QPSK, 16QAM, whatever) longer distance link, be absolutely
sure to poll all of the SNMP OIDs for it the same as if it was a point to
point microwave link.

Depending on exactly what line card and optic it is, it may behave somewhat
similarly to a faded or misaligned radio link under conditions related to
degradation of the fiber or the lasers. In particular I'm thinking of
coherent 100G linecards that can switch on the fly between 'low FEC' and
'high FEC' payload vs FEC percentage (much as an ACM-capable 18 or 23 GHz
band radio would), which should absolutely trigger an alarm. And also the
data for FEC decode stress percentage level, etc.

On Thu, Apr 29, 2021 at 2:37 PM Lady Benjamin Cannon of Glencoe, ASCE <
l...@6by7.net> wrote:

> We monitor light levels and FEC values on all links and have thresholds
> for early-warning and PRe-failure analysis.
>
> Short answer is yes we see links lose packets before completely failing
> and for dozens of reasons that’s still a good thing, but you need to
> monitor every part of a resilient network.
>
> Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
> 6x7 Networks & 6x7 Telecom, LLC
> CEO
> l...@6by7.net
> "The only fully end-to-end encrypted global telecommunications company
> in the world.”
>
> FCC License KJ6FJJ
>
> Sent from my iPhone via RFC1149.
>
> On Apr 29, 2021, at 2:32 PM, Eric Kuhnke  wrote:
>
> 
> The Junipers on both sides should have discrete SNMP OIDs that respond
> with a FEC stress value, or FEC error value. See blue highlighted part here
> about FEC. Depending on what version of JunOS you're running the MIB for it
> may or may not exist.
>
>
> https://kb.juniper.net/InfoCenter/index?page=content&id=KB36074&cat=MX2008&actp=LIST
>
> In other equipment sometimes it's found in a sub-tree of SNMP adjacent to
> optical DOM values. Once you can acquire and poll that value, set it up as
> a custom thing to graph and alert upon certain threshold values in your
> choice of NMS.
>
> Additionally signs of a failing optic may show up in some of the optical
> DOM MIB items you can poll:
> https://mibs.observium.org/mib/JUNIPER-DOM-MIB/
>
> It helps if you have some non-misbehaving similar linecards and optics
> which can be polled during custom graph/OID configuration, to establish a
> baseline 'no problem' value, which if exceeded will trigger whatever
> threshold value you set in your monitoring system.
>
> On Thu, Apr 29, 2021 at 1:40 PM Baldur Norddahl 
> wrote:
>
>> Hello
>>
>> We had a 100G link that started to misbehave and caused the customers to
>> notice bad packet loss. The optical values are just fine but we had packet
>> loss and latency. Interface shows FEC errors on one end and carrier
>> transitions on the other end. But otherwise the link would stay up and our
>> monitor system completely failed to warn about the failure. Had to find the
>> bad link by traceroute (mtr) and observe where packet loss started.
>>
>> The link was between a Juniper MX204 and Juniper ACX5448. Link length 2
>> meters using 2 km single mode SFP modules.
>>
>> What is the best practice to monitor links to avoid this scenarium? What
>> options do we have to do link monitoring? I am investigating BFD but I am
>> unsure if that would have helped the situation.
>>
>> Thanks,
>>
>> Baldur
>>
>>
>>

Re: Call for academic researchers (Re: New minimum speed for US broadband connections)

[Eric Kuhnke]

An interesting question would be to quantify and do statistical analysis on
the following:

Take a set of 1000 or more residential last mile broadband customers on an
effectively more-than-they-can-use connection (symmetric 1Gbps active
ethernet or similar).

On a 60s interval, retrieve SNMP traffic stats from the interfaces towards
the customers' demarcs, or directly from on premises CPEs.

Store that data in influxdb or another lossless time series database for a
multi month period.

Anonymize the data so that no possible information about the
identity/circuit ID/location of the customer can be identified. Perhaps
other than "gigE customer somewhere in North America", representing a semi
random choice of US/Canada domestic market residential broadband users.

Provide that data set to persons who wish to analyze it to see how much/how
bursty the traffic really is, night/day traffic patterns, remote work
traffic patterns during office hours in certain time zones, etc.
Additionally quantify what percentage of users move how much upstream data
or come anywhere near maxing it out in brief bursts (people doing full disk
offsite backups of 8TB HDDs to Backblaze, uploading 4K videos to youtube,
etc).

I at first thought of a concept of doing something similar but with netflow
data on a per CPE basis, but that has a great deal more worrisome privacy
and PII data implications than simply raw bps/s interface data. Presumably
netflow (or data from Kentik, etc) for various CDN traffic and other per-AS
downstream traffic headed to an aggregation router that serves exclusively
a block of a few thousand downstream residential symmetric gigabit
customers would not be a difficult task to sufficiently anonymize.




On Sat, May 29, 2021 at 4:25 PM Sean Donelan  wrote:

>
> I thought in the 1990s, we had moved beyond using average bps measurements
> for IP congestion collapse.  During the peering battles, some ISPs used to
> claim average bps measurements showed no problems.  But in reality there
> were massive packet drops, re-transmits and congestive collapse which
> didn't show up in simple average bps graphs.
>
>
> Have any academic researchers done work on what are the real-world minimum
> connection requirements for home-schooling, video teams applications, job
> interview video calls, and network background application noise?
>
>
> During the last year, I've been providing volunteer pandemic home
> schooling support for a few primary school teachers in a couple of
> different states.  Its been tough for pupils on lifeline service (fixed
> or mobile), and some pupils were never reached. I found lifeline students
> on mobile (i.e. 3G speeds) had trouble using even audio-only group calls,
> and the exam proctoring apps often didn't work at all forcing those
> students to fail exams unnecessarily.
>
> In my experience, anecdotal data need some academic researchers, pupils
> with at least 5 mbps (real-world measurement) upstream connections at
> home didn't seem to have those problems, even though the average bps graph
> was less than 1 mbps.
>
>

Re: Call for academic researchers (Re: New minimum speed for US broadband connections)

[Eric Kuhnke]

If one installs smokeping on a raspberry pi using a wired ethernet
interface to a home router, on a DOCSIS3 residential last mile segment, and
copies over a well chosen targets file for things to test, and sets it to a
60s interval, all other settings at default...  It's quite rare to find a
network segment that isn't anywhere from 0.05 to 0.30% packet loss (or
sometimes worse!) to its default gateway over a 24 hour period.

Also very informative is when you see spikes in latency and jitter during
evening peak usage hours. Lots of very basic test methodology can reveal
the nature of oversubscribed contended access mediums. Last-mile 5 GHz band
PtMP WISPs can see exactly the same issues on an overloaded AP.





On Mon, May 31, 2021 at 12:09 PM Denys Fedoryshchenko <
nuclear...@nuclearcat.com> wrote:

> It can't be zero.
> In 1000BaseT specs, BER, 1 in 1*10^10 bits error is considered
> acceptable on each link.
> So it should be defined same way, as acceptable BER.
> And until which point? How to measure?
> Same for bandwidth, port rate can be 1Gbit, ISP speedtest too, but most
> websites 100Kbit.
>
> On 2021-05-31 21:28, Fred Baker wrote:
> > I would add packet loss rate. Should be zero, and if it isn’t, it
> > points to an underlying problem.
> >
> > Sent from my iPad
> >
> >> On May 31, 2021, at 11:01 AM, Josh Luthman
> >>  wrote:
> >
> >> 
> >> I think the latency and bps is going to be the best way to measure
> >> broadband everyone can agree on.  Is there a better way, sure, but
> >> how can you quantify it?
> >>
> >> Josh Luthman
> >> 24/7 Help Desk: 937-552-2340
> >> Direct: 937-552-2343
> >> 1100 Wayne St
> >> Suite 1337
> >> Troy, OH 45373
> >>
> >> On Sun, May 30, 2021 at 7:16 AM Mike Hammett 
> >> wrote:
> >>
> >>> I think that just underscores that the bps of a connection isn't
> >>> the end-all, be-all of connection quality. Yes, I'm sure most of
> >>> us here knew that. However, many of us here still get distracted
> >>> by the bps.
> >>>
> >>> If we can't get it right, how can we expect policy wonks to get it
> >>> right?
> >>>
> >>> -
> >>> Mike Hammett
> >>> Intelligent Computing Solutions
> >>> http://www.ics-il.com
> >>>
> >>> Midwest-IX
> >>> http://www.midwest-ix.com
> >>>
> >>> -
> >>>
> >>> From: "Sean Donelan" 
> >>> To: "NANOG" 
> >>> Sent: Saturday, May 29, 2021 6:25:12 PM
> >>> Subject: Call for academic researchers (Re: New minimum speed for
> >>> US broadband connections)
> >>>
> >>> I thought in the 1990s, we had moved beyond using average bps
> >>> measurements
> >>> for IP congestion collapse.  During the peering battles, some ISPs
> >>> used to
> >>> claim average bps measurements showed no problems.  But in reality
> >>> there
> >>> were massive packet drops, re-transmits and congestive collapse
> >>> which
> >>> didn't show up in simple average bps graphs.
> >>>
> >>> Have any academic researchers done work on what are the real-world
> >>> minimum
> >>> connection requirements for home-schooling, video teams
> >>> applications, job
> >>> interview video calls, and network background application noise?
> >>>
> >>> During the last year, I've been providing volunteer pandemic home
> >>> schooling support for a few primary school teachers in a couple of
> >>>
> >>> different states.  Its been tough for pupils on lifeline service
> >>> (fixed
> >>> or mobile), and some pupils were never reached. I found lifeline
> >>> students
> >>> on mobile (i.e. 3G speeds) had trouble using even audio-only group
> >>> calls,
> >>> and the exam proctoring apps often didn't work at all forcing
> >>> those
> >>> students to fail exams unnecessarily.
> >>>
> >>> In my experience, anecdotal data need some academic researchers,
> >>> pupils
> >>> with at least 5 mbps (real-world measurement) upstream connections
> >>> at
> >>> home didn't seem to have those problems, even though the average
> >>> bps graph
> >>> was less than 1 mbps.
>

Re: New minimum speed for US broadband connections

[Eric Kuhnke]

Perhaps you may be unfamiliar with the business model of cities, counties
or local PUDs running the fiber last mile network (at OSI layer 1) and
providing ethernet transport/VLAN handoffs, installing the OLTs and ONTs,
and third party ISPs using that network to provide IP, support, billing and
over-the-top services riding on it. In some of these cases the PUD is also
the entity which operates the last mile electrical distribution network,
and can put fiber on their own poles, which greatly reduces costs and
speeds up deployment.

The customers of the half dozen PUDs in eastern WA which use this business
model are presently enjoying gigabit class residential access at around
$50-75/month and are very happy with it.





On Mon, May 31, 2021 at 10:58 AM Josh Luthman 
wrote:

> I think it's hilarious when a governmental entity funded by the taxpayers
> thinks they have an answer to broadband.  If you're collecting funds from
> customers, why do you need the City of Sherwood to support your network?
>
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
>
> On Fri, May 28, 2021 at 6:22 PM Brandon Price 
> wrote:
>
>> 100/100 minimum for sure.
>>
>> In our small neck of the woods, we are currently doing 250/250 for $45
>> and 1000/1000 for $60 no data caps.
>>
>> We have lost some grants on rural builds because "someone" in the census
>> block claims they provide broadband.. Not hard to put an AP up on a tower
>> and hit the current definition's upload speed.
>>
>> I get a chuckle when the providers tell the customer what they "need"...
>>
>>
>> Brandon Price
>> Senior Network Engineer
>> City of Sherwood, Sherwood Broadband
>>
>>
>>
>> -Original Message-
>> From: NANOG  On
>> Behalf Of Sean Donelan
>> Sent: Thursday, May 27, 2021 5:33 PM
>> To: NANOG Operators' Group 
>> Subject: Re: New minimum speed for US broadband connections
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you are expecting this email and/or
>> know the content is safe.
>>
>>
>> On Thu, 27 May 2021, Lady Benjamin Cannon of Glencoe, ASCE wrote:
>> > At least 100/100.
>> >
>> > We don’t like selling slower than 10g anymore, that’s what I’d start
>> everyone at if I could.
>>
>>
>> At $50/month or less?
>>
>> Maximize number of households of all demographic groups.
>>
>>

Re: New minimum speed for US broadband connections

[Eric Kuhnke]

I think it has been true for many years that:

a) a vast majority of residential gigabit/symmetric customers, or gigabit
asymmetric (docsis3 500-1000 down, 16-50 up) no longer have a device in
their home with a 1000BaseT port on it, or don't know if they do. in some
cases literally the only cat5e cable they have may be a 3' piece from their
cable modem to 'router', and everything else is wifi.

b) don't understand the difference between the service speed delivered via
the wireline connection and demarc/handoff device, whatever that may be,
and their perception of service over the wifi.

c) are unwilling to go through troubleshooting steps requiring them to
directly connect a device to the modem/demarc by 1000BaseT and run speed
tests, possibly necessitating a service call (this can be partially avoided
by the install technician doing a *wired* speed test in front of the
customer at the time of install, from their laptop, and taking a couple of
minutes to explain the difference)

d) may be using badly configured wifi things that stomp on each other,
sometimes provided by the ISP (I have seen set-top boxes from major MSOs
that broadcast a 2x2 MIMO 802.11ac 80 MHz wide channel, now imagine ten of
these all in wood framed houses/condos/townhouses all very close to each
other, in addition to the wifi from the demarc modem/router device). There
are lots of other things in the common consumer environment that render
some environments a CSMA mishmash, like smart TVs, printers and things that
all create their own AP for some reason.

e) may be using their own randomly purchased-from-best-buy wifi "range
extender" devices to create weird forms of mesh networks in their home,
further halving their bandwidth with each half duplex hop.





On Mon, May 31, 2021 at 4:54 PM Tim Burke  wrote:

> This is a good point as well… you can have the largest pipe in the world,
> but in many cases, in-home service issues are caused by crappy CPE.
>
>
>
> Example… my neighborhood has 1000/50 GPON (rather silly to offer such poor
> upload speed, but that’s irrelevant in this case) provided by a local
> outfit, Entouch (now Grande/RCN) as part of HOA dues… Many people in the
> neighborhood do not use it and blame the ISP for offering “mediocre
> service”, simply because there is no fancy CPE included as part of the
> service offering. Yet as soon as you swap that $25 Netgear router
> pre-installed by the home builder’s structured wiring contractor for
> something that’s worth a damn, the pipe is actually usable…
>
>
>
> With that said, if there needs to be regulation on minimum broadband
> speeds, should there be regulation to require home ISPs to provide high-end
> 802.11ax-capable network gear, so the average clueless home user with a
> 1gbps FTTP connection can actually use the service they’re paying for?
>
>
>
> V/r
>
> Tim
>
>
>
> *From:* NANOG  * On Behalf Of *Josh
> Luthman
> *Sent:* Monday, May 31, 2021 12:55 PM
> *To:* NANOG list 
> *Subject:* Re: New minimum speed for US broadband connections
>
>
>
> Was that the fault of the broadband provider or was that the fault of the
> indoor WiFi?  Is it possible the router has so much interference from all
> of the neighbors and everyones using 2.4 GHz?  What if that example had a
> cable connection with 960/40 mbps and they're limited to 5 mbps up because
> of the in house WiFi solution?
>
> Would upping the broadband plan to 1000/1000 fix that problem?
>
>
>
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
>
>
>
>
> On Fri, May 28, 2021 at 2:56 PM Chris Adams  wrote:
>
> Once upon a time, Mike Hammett  said:
> > "Bad connection" measures way more than throughput.
> >
> > What about WFH or telehealth doesn't work on 25/3?
>
> More than one person in a residence, home security systems (camera,
> doorbell, etc.) uploading continuously, and more.
>
> I know multiple people that had issues with slow Internet during the
> last year as two adults were working from home and 1-3 children were
> also schooling from home.  Parents had to arrange work calls around
> their kids classroom time and around each other's work calls, because of
> limited bandwidth.
>
> The time of the Internet being a service largely for consumption of data
> is past.  While school-from-home may be a passing thing as the pandemic
> wanes, it looks like work-from-home (at least part time) is not going to
> go away for a whole lot of people/companies.
>
> --
> Chris Adams 
>
>

Re: New minimum speed for US broadband connections

[Eric Kuhnke]

Perhaps there should be some sort of harsher penalty for ILECs and other
large near-monopoly last mile local carriers that outright lie on their
form 477 data or take significant subsidy funds and then fail to build what
they promised. Numerous states' attorney generals have gone after them on
this matter to varied degrees of success.



On Mon, May 31, 2021 at 5:27 PM Mike Hammett  wrote:

> No one's paying me anything except 15 years of practical experience
> building last mile networks for myself and my clients. I'd imagine that
> while a larger percentage than most venues, a minority of the people on
> this list build last mile networks. Even fewer do so with their own money.
>
> I have a fiber network where I offer gigabit bidirectional to the home.
>
>
> Few people have any sort of grasp of the cost and complexity of building
> what they want.
>
> Raising the the minimal definitions for everyone to what power users
> expect is a foolish venture.
>
>
>
>
> I'm just trying to connect some of you to reality.
>
>
> " Doesn’t matter." Yes, it matters very much so when you're proposing the
> expenditure of my money to meet your unrealistic goals. I'm not against
> raising the definition. I'm not against offering 1G or 10G to the home. I'm
> against you telling people that are perfectly happy with their service that
> it's not good enough for them and then using their and my tax dollars to
> "fix" it.
>
>
>
> I don't disagree that the big ISPs have screwed the pooch many times and
> will do so in perpetuity. These programs often just give those same
> entities that screwed us all for years the money to do it. That's partially
> why they don't spend their own money doing it. They'll wait for Uncle Sam
> to pay them to do it.
>
>
> Muni broadband does suck, but that's another thread for another day.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Andy Ringsmuth" 
> *To: *"NANOG" 
> *Sent: *Monday, May 31, 2021 9:17:17 AM
> *Subject: *Re: New minimum speed for US broadband connections
>
> As much as I enjoy the generally cordial nature of this list, I’m going to
> go out on a limb and say that Mr. Hammett’s mentality on this topic is
> precisely the problem. Arguing against every reasonable proposition we are
> making to increase home broadband speeds.
>
> I’m assuming he’ll disagree. And that’s OK. He’s still wrong.
>
> “People want X. Why?”  - Doesn’t matter. I don’t need a reason for what I
> want. I probably have one, but that reason is my business, not yours.
>
> The big ISPs are, historically and factually, greedy, stingy, and in many
> cases flat-out liars on all this. Taking USF money for DECADES and
> squandering it, for instance. Advertising speeds (I’m looking at you,
> Frontier) they knew full well they couldn’t provide. Charging $40 for
> service on one street and $80 for IDENTICAL service one block away.
> Promising to state governments they would upgrade and then not doing it
> (Charter in New York, anyone?).
>
> Blah blah blah shareholders blah blah blah. DGAF.
>
> Where there is a will, there is a way. The big boys don’t have the will to
> do it. Case after case after case after case after case demonstrates that
> fiber to the home can be done and can be done for a very reasonable cost.
> We read about smaller companies or municipalities every day doing it. And
> then the Big Boys come along and do EVERYTHING they can to stifle
> competition (getting all snarky about pole access, or pouring billions into
> lobbying against muni broadband that could be spent on, oh, I dunno,
> INSTALLING FIBER instead).
>
> “When making policy changes and spending hundreds of billions of dollars,
> you need to have a good reason.” Apply that same thinking to all the
> reasons the Big Boys give for NOT installing fiber or upgrading their
> networks. How many billions have they spent on lobbying and lawsuits to
> stop competition and not install fiber that could have been better spent?
>
> I will go so far as to directly ask:
>
> Mike - who is paying you to lobby so hard against better/faster/more
> reliable home internet?
>
> 
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> a...@andyring.com
>
> “Better even die free, than to live slaves.” - Frederick Douglas, 1863
>
> > On May 31, 2021, at 8:01 AM, Mike Hammett  wrote:
> >
> > Why is any of that a reasonable position to have? What you're proposing
> is reckless without real, compelling evidence.
> >
> > People want X. Why?
> >
> > When making policy changes and spending hundreds of billions of dollars,
> you need to have a good reason.
> >
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> > From: "Baldur Norddahl" 
> > To: "NANOG" 
> > Sent: Sunday, May 30, 2021 12:53:25 PM
> > Subject: Re: New min

OSI layer 1 and revisiting labelmakers in the year 2021

[Eric Kuhnke]

 I am still using a Dymo 4200 [1] which is generally okay. I am wondering
if anyone or their field tech team has recently changed to a better label
maker in terms of feature set, battery life/charging or label consumable
cost.

Surely there must be something better out there. Strong preference for
QWERTY keyboards, no ABCDE type.

[1]: https://www.dymo.com/en_CA/rhino-industrial-4200-qwy.html

Re: Google uploading your plain text passwords

[Eric Kuhnke]

I think you have only found the tip of the iceberg of things that Chrome
and Google does without your express consent.

On Fri, Jun 11, 2021 at 9:48 AM William Herrin  wrote:

> On Fri, Jun 11, 2021 at 9:38 AM Jan Schaumann via NANOG 
> wrote:
> > William Herrin  wrote:
> > > It turns out that every password I allowed Chrome on Android to
> > > remember, it uploaded to Google. In plain text!!
> >
> > Chrome does not store your passwords in plain text.
> > It encrypts them locally, on e.g. macOS using, I
> > think, a secret stored in the keychain under "Chrome
> > Safe Storage", on Windows using a similar API and
> > secret probably unlocked via your login credentials.
>
> Hi Jan,
>
> I'm fine with Chrome encrypting them locally. That's what I want it to
> do. I'm not at all fine with it uploading them to my Google account. I
> don't want any trace of my non-google passwords present in my google
> account. I'm very very not fine that it happened behind my back
> without my express consent.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>