Re: Hurricane Electric AS6939

2020-10-14 Thread Jared Mauch



> On Oct 13, 2020, at 7:29 PM, Aaron Gould  wrote:
> 
> Do y’all like HE for Internet uplink?  I’m thinking about using them for 
> 100gig in Texas.  It would be for my eyeballs ISP.  We currently have 
> Spectrum, Telia and Cogent.

No.  Too many problems with scenic routing due to lack of BGP communities to 
control route advertisements.

Also the usual lack of full table issue due to route suppression on their 
transits based on AFI.

I’m sure it works well for many people, but if you are large enough you need to 
do any form of TE the lack of BGP communities is a major issue.

- Jared

Re: Ingress filtering on transits, peers, and IX ports

2020-10-14 Thread Jared Mauch
On Tue, Oct 13, 2020 at 05:49:42PM -0500, Brian Knight via NANOG wrote:
> Hi Mel, 
> 
> My understanding of uRPF is: 
> 
> * Strict mode will permit a packet only if there is a route for the
> source IP in the RIB, and that route points to the interface where the
> packet was received 
> 
> * Loose mode will permit a packet if there is a route for the source IP
> in the RIB.  It does not matter where the route is pointed. 
> 
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit. 
> 
> Loose mode won't work for us, because all of our own prefixes are in our
> RIB, and thus the uRPF check on a transit would never block anything. 

You'll be surprised at the garbage you would drop that you can't return.

- Jared


Re: Apple Catalina Appears to Introduce Massive Jitter

2020-10-29 Thread Jared Mauch
On Thu, Oct 29, 2020 at 02:31:59PM +0200, Mark Tinka wrote:
> 
> 
> On 10/29/20 14:27, Matt Hoppes wrote:
> 
> > Is it actually jitter or is it potentially the wireless network card
> > going into sleep mode? I have seen that type of behavior on Apple
> > products when the cards go into low power mode although I can’t say I
> > have noticed that on my laptop.
> 
> Not, not sleep mode, because I am actively using the device to move data
> to/from the Internet.
> 
> I was actually struggling to upload some files to Youtube last weekend, and
> had to use another computer to do it as I couldn't figure out what was going
> on. That it is how bad the jitter is.
> 
> It seems to be a much bigger problem for the upload direction than the
> download, but it, inevitably creates a symmetrical performance problem.

I know there was a recent fix Apple did for devices talking to UBNT APs
for their handsets, perhaps there's a similar fix needed on your side?

I have all UBNT at home for wireless and periodically have some random
issues which I can't explain, but for the most part have things tuned to ensure
there's little to no interference.

Do you see the same when hardwired?  I keep many of my devices hardwired
to avoid odd jitter issues.  I also saw some older versions of the Pulse Secure
VPN add the behavior you describe, including the more uptime the slower it would
get.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: Weather Service faces Internet bandwidth shortage, proposes limiting key data

2020-12-10 Thread Jared Mauch



> On Dec 10, 2020, at 9:39 AM, Keith Medcalf  wrote:
> 
> 
> Simply get rid of the gigabytes of JavaScript and stupidly designed crap
> and hire someone who knows what they are doing and a bandwidth DOWNGRADE
> will be in order.  The root cause is incompetence and it can be fixed by
> getting rid of all the children and hiring someone who knows what they
> are doing.
> 


If the only problem was bandwidth it would be easy to solve.

I miss weather underground before it became slow as molasses with openstreetmap 
and other things.  Then again, it used to be a local telephone call and typing 
UM-WEATHER at the which host? prompt

- Jared

Re: Need someone with clue @ Network Solutions.

2020-12-15 Thread Jared Mauch
Matthew,

I haven’t seen this problem in a long time where someone else submits data to 
cause the out-of-zone glue to appear.  It’s possible there’s something 
happening at NETSOL that is causing this, but the best way is for you to go 
into your registrar and ensure they’re publishing the proper host records for 
your in-zone glue which should address this if nobody got back to you yet.  It 
may also be easier to find someone on the dns-operations list than NANOG these 
days.

- Jared

> On Dec 15, 2020, at 11:43 AM, Matthew Crocker  
> wrote:
> 
> I need to get Network Solutions to remove glue records for hosts in my 
> domain.   My domain isn’t registered with Network Solutions and they refuse 
> to speak with me as I’m not a customer.
>  
> I’ve had my customer attempt to update their domain through Network Solutions 
> but the only thing they can change is the NS record, not the underlying host 
> glue record.   I don’t think the glue records even need to exist as they are 
> published by my domain already.
>  
> Does anyone have any contacts at Network Solutions that can help?
>  
> Example:
>  
> dig .com NS @i.gtld-servers.net.
>  
> ; <<>> DiG 9.10.6 <<>> .com NS @i.gtld-servers.net.
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24593
> ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
> ;; WARNING: recursion requested but not available
>  
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;.com.IN NS
>  
> ;; AUTHORITY SECTION:
> .com.  172800 IN NS dns-auth4.crocker.com.
> .com.  172800 IN NS dns-auth3.crocker.com.
>  
> ;; ADDITIONAL SECTION:
> dns-auth4.crocker.com.  172800 IN A  66.59.48.95
> dns-auth3.crocker.com.  172800 IN A  66.59.48.94
>  
> ;; Query time: 73 msec
> ;; SERVER: 192.43.172.30#53(192.43.172.30)
> ;; WHEN: Tue Dec 15 11:34:41 EST 2020
> ;; MSG SIZE  rcvd: 124
>  
> The correct servers are:
>  
> dns-auth3.crocker.com.  299IN A  66.59.61.10
> dns-auth4.crocker.com.  299IN A  66.59.61.194



Re: 10g residential CPE

2020-12-25 Thread Jared Mauch



> On Dec 25, 2020, at 12:48 PM, Bryan Fields  wrote:
> 
> 10g to the home is a great idea to think about, it's just not terribly
> practical for most customers unless they want to drop 1-2k on routing gear and
> nics.  This is always changing, but it's going to be a few years until we
> reach the right performance and price point.

Think more using your PON network to also serve commercial customers so you 
don't need high end CPE to hit 1-5Gbps or WDM setups. 

Re: [External] 10g residential CPE

2020-12-28 Thread Jared Mauch



> On Dec 25, 2020, at 5:32 PM, John Levine  wrote:
> 
> I agree it is odd to make 100/100 the top speed. The fiber service I
> have from my local non-Bell telco offers 100/100, 500/500, and
> 1000/1000. FiOS where you can get it goes to 940/880.
> 
> The obvious guess is that their upstream bandwidth is
> underprovisioned, or maybe they figure 100/100 is all they need to
> compete in that particular market.

My TV (wired) pulls at higher bitrates when doing the initial fetches of the 
buffering.  Not unusual to see it pulling more than 150Mb/s at the start of a 
(non-4K) show.

I think the extent that end-users are impacted by these slower speeds while 
buffering is under appreciated in the experience.

At $dayjob many servers are 10G or 100G so the limiting factor is most likely 
the CPE or ISP.  I was hearing last night about someone with a device that 
didn’t appear to be hitting the line-rate but was dropping 0.5% of packets when 
running at 3Gb/s until they upgraded to one of the major networking vendors we 
all know here.

In my small FTTH network the slowest link is at the customer home and all the 
devices are hardware ASIC forwarded vs offload as you find in some of the 
low/mid-tier devices (eg: Tik/UBNT).

Many streaming things do 8 second waits between chunks, so if you’re pulling a 
video stream at 6Mb/s you really are pulling 6*8 (lets say 50) then idle for 7 
seconds.  If you’re on a 25Mb/s service or even a 50Mb/s service it won’t work 
the way you expect if there’s any other activity.

- Jared

Re: 10g residential CPE

2020-12-28 Thread Jared Mauch



> On Dec 26, 2020, at 10:35 AM, Mikael Abrahamsson via NANOG  
> wrote:
> 
> Here the "truth" is that if you game, you need to have a wired connection to 
> your gaming computer. All gamers "know" this.

My sons switch is hard wired, he gets considerable advantage (apparently) due 
to using the USB adapter vs wifi when playing online.

- Jared

advertise-peer-as

2021-01-16 Thread Jared Mauch
I’m curious how many networks have this as part of their default configuration?

If you’re bored on a weekend please click through to this and say yes/no.  I’ll 
summarize in a week or so.  If I configured the form right, it should also show 
you the results as well.

https://forms.gle/xxLnA7KgQL48VMNe8

- Jared

Summary: advertise-peer-as

2021-01-27 Thread Jared Mauch


I’ve closed the form for responses.

There were 102 people who self-selected to participate. 

Around 25% of you have this configured as a default in your network.

At 20940 we are asking our networking partners to configure this facing us such 
that we can receive the routes from some of the stub networks we operate.

I was also told there’s some interesting behavior in IOS-XR, if a peer is in 
the same peer-group and based on the order of the peers coming up, the behavior 
may be different where routes may be suppressed.

You may also want to read up on the "send-transit-peer” command for your 
network.

Thanks for responding, and if you have a BGP peer with 20940 please set 
advertise-peer-as or similar.  It may help keep more traffic on direct links as 
a result.

- jared

Re: New York Carrier Hotels

2021-02-15 Thread Jared Mauch
I’m expecting many people to move out to 165 Halsey but as with many things the 
future is still hazy.  I also wonder if at some point Google will decide that 
WFH is viable and they don’t need the office space in 111 8th and things will 
swing back..

(Yes, I know that 165 isn’t in NY)

- Jared

> On Feb 11, 2021, at 1:51 PM, Rod Beck  wrote:
> 
> Hey Folks, 
> 
> I am looking for a list of the ten most important NYC telecom hotels. Over 
> the last 15 years carrier business has shifted to a large extent to Secaucus 
> Equinix & Google has taken over a big part of 111 8th Avenue. What the 
> important sites today and are any new facilities on the horizon?
> 
> Roderick Beck
> Global Network Capacity Procurement
> United Cable Company
> www.unitedcablecompany.com
> https://unitedcablecompany.com/video/
> New York City & Budapest
> rod.b...@unitedcablecompany.com
> Budapest: 36-70-605-5144
> NJ: 908-452-8183 



Re: Texas internet connectivity declining due to blackouts

2021-02-16 Thread Jared Mauch
Almost exactly 4 years ago we were out up here in Michigan for over 120 hours 
after a wind storm took out power to 1 million homes. Large scale restoration 
takes time. When the load and supply are imbalanced it can make things worse as 
well. 

I'm hoping things return to normal soon but also am reminded it can take some 
time. 

We now have a large generator with automatic switchover after that event. 
Filling gas cans every 12 hours to feed the generator is no fun. 

Sent from my TI-99/4a

> On Feb 15, 2021, at 11:54 PM, Cory Sell via NANOG  wrote:
> 
>  Total population is a pretty big difference as you go north, as is how well 
> infrastructure is actually prepared for snow/ice and cold temperatures in 
> general.
> 
> I’ve been without power all day and have no doubt I’ll cross the 24-hour mark 
> here in a handful of hours.
> 
> Sent from ProtonMail Mobile
> 
> 
>> On Mon, Feb 15, 2021 at 10:42 PM, Sean Donelan  wrote:
>> 
>> 
>> On Tue, 16 Feb 2021, Cory Sell via NANOG wrote:
>> > adoption. Sure, wind isn’t perfect, but looks like solution relied on 
>> > failed
>> > in a massive way.
>> 
>> Strange the massive shortages and failures are only in one state.
>> 
>> The extreme cold weather extends northwards across many states, which
>> aren't reporting rolling blackouts.
> 
> 


Re: Texas internet connectivity declining due to blackouts

2021-02-16 Thread Jared Mauch



> On Feb 16, 2021, at 8:25 AM, Mike Hammett  wrote:
> 
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
> underground plant.
> 
> The actual cost per foot is more like 10x difference, but there are right of 
> way, maintenance, etc. costs to factor in as well.
> 

Labor is something in 8x but the permit/engineering cost is usually the same 
per foot, but the make-ready on poles can make underground competitive or more 
like 1.5x when you fully bake the costs.  Really depends on pole distances and 
quality.  O&M long-term is lower on underground vs aerial.

- Jared

Re: Famous operational issues

2021-02-16 Thread Jared Mauch
I was thinking about how we need a war stories nanog track. My favorite was 
being on call when the router was stolen. 

Sent from my TI-99/4a

> On Feb 16, 2021, at 2:40 PM, John Kristoff  wrote:
> 
> Friends,
> 
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?
> 
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
> 
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
> 
> Thanks in advance for your suggestions,
> 
> John


Re: Famous operational issues

2021-02-17 Thread Jared Mauch
The he.net side is interesting as you can see who their v4 transits are but 
they suppress their routes via v6, but (last I knew) lacked community support 
for their customers to do similar route suppression.

I’m not a fan of it, but it makes the commercial discussions much easier each 
time those networks come by to shop services to me in a personal or 
professional capacity.  “No, I need all the internet”.

- Jared

> On Feb 17, 2021, at 12:07 PM, David Guo via NANOG  wrote:
> 
> Cogentco still did not peer with Google and HE over IPv6 I guess.
> 
> From: NANOG  on behalf of Justin 
> Wilson (Lists) 
> Sent: Thursday, February 18, 2021 00:53
> To: Miles Fidelman
> Cc: nanog@nanog.org
> Subject: Re: Famous operational issues
>  
> I remember when the big carriers de-peered with Cogent in the early 2000s.  
> The underestimated the amount of web-sites being hosted by people using 
> cogent exclusively. 
> 
> 
> Justin Wilson
> j...@j2sw.com
> 
> —
> https://j2sw.com - All things jsw (AS209109)
> https://blog.j2sw.com - Podcast and Blog
> 
> > On Feb 17, 2021, at 10:29 AM, Miles Fidelman  
> > wrote:
> > 
> > John Kristoff wrote:
> >> Friends,
> >> 
> >> I'd like to start a thread about the most famous and widespread Internet
> >> operational issues, outages or implementation incompatibilities you
> >> have seen.
> >> 
> > Well... pre-Internet, but the great Northeast fiber cut comes to mind 
> > (backhoe vs. fiber, backhoe won).
> > 
> > Miles Fidelman
> > 
> > -- 
> > In theory, there is no difference between theory and practice.
> > In practice, there is.   Yogi Berra
> > 
> > Theory is when you know everything but nothing works. 
> > Practice is when everything works but no one knows why. 
> > In our lab, theory and practice are combined: 
> > nothing works and no one knows why.  ... unknown



Re: Famous operational issues

2021-02-18 Thread Jared Mauch
On Thu, Feb 18, 2021 at 01:07:01AM -0800, Eric Kuhnke wrote:
> On that note, I'd be very interested in hearing stories of actual incidents
> that are the cause of why cardboard boxes are banned in many facilities,
> due to loose particulate matter getting into the air and setting off very
> sensitive fire detection systems.
> 
> Or maybe it's more mundane and 99% of the reason is people unpack stuff and
> don't always clean up properly after themselves.

We had a plastic bag sucked into the intake of a router in a
datacenter once that caused it to overheat and take the site down.  We
had cameras in our cage and I remember seeing the photo from the site of
the colo (I'll protect their name just because) taken as the tech was on
the phone and pulled the bag out of the router.

The time from the thermal warning syslog that it's getting warm
to overheat and shutdown is short enough you can't really get a tech to
the cage in time to prevent it.

I assume also the latter above, which is people have varying
definitons of clean.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: Trouble Playing Multiplayer Games from Reallocated IP Space

2021-03-03 Thread Jared Mauch
Tim, I'll get you off-list.

- Jared

On Wed, Mar 03, 2021 at 10:37:44AM -0500, Tim Nowaczyk wrote:
> Hey NANOG,
> 
> We have seen an issue where our customers who have IP addresses that are 
> directly allocated to us can play online multiplayer games (NBA2k, NBA2k21, 
> Fallout 76, and Stardew Valley were mentioned specifically) but when they 
> have an IP that was reallocated to us by one of our Upstreams (Server 
> Central), these games no longer work. I know NBA2k is behind Prolexic, but 
> not sure about the others. 2k Games says that we, the ISP, must be blocking 
> something, but there’s absolutely no difference in how data moves from our 
> internet edge to the customer whether they have one of our IPs or one of the 
> reallocated ones. We noticed that one geocoding provider has our reallocated 
> IPs flagged as “hosting” IPs and maybe 2k is using some metadata like that to 
> block our IPs.
> 
> Does anyone have a contact at Prolexic who might be able to help?
> 
> Thanks,
> Tim Nowaczyk
> 
> --  
> Timothy Nowaczyk  |  Senior Network Manager
> office  703.554.6622   |  mobile  571.318.9434 
> 

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: wow, lots of akamai

2021-04-01 Thread Jared Mauch



> On Apr 1, 2021, at 11:15 AM, Töma Gavrichenkov  wrote:
> 
> Peace,
> 
> On Thu, Apr 1, 2021, 6:09 PM  wrote:
> That was a lot of traffic coming out of akamai aanp clusters the last couple 
> nights!  What was it?
> 
> "Call of Duty" update again, obviously.
> 
> https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzone-update-is-over-50gb


Yes, we have had a number of big customer traffic in recent days.  Hopefully 
traffic is flowing well for many networks.  If this is negatively impacting 
you, please reach out.

- Jared

Re: wow, lots of akamai

2021-04-02 Thread Jared Mauch
On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote:
> On 4/1/2021 3:21 PM, Niels Bakker wrote:
> > * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> > CEST]:
> >> An artificial roll out penalty somehow? Probably not at the ISP level,
> >> but more at the game level. Well, ISP could also have some mechanisms
> >> to reduce the impact or even Akamai could force a progressive roll out.
> > 
> > It's an online game. You can't play the game with outdated assets. You'd
> > not see walls where other players would, for example.
> > 
> > What you're suggesting is the ability of ISPs to market Internet access
> > at a certain speed but not have to deliver it based on conditions they
> > create.
> > 
> > 
> > -- Niels.
> 
> 
> There was at least one online gaming platform that used to stage their
> game updates, where clients would download the update over the previous
> week or so, and then on go-live day, the game updated itself.  There
> were a few that had to download on go-live day, but the clients I ran,
> and friends ran, had downloaded prior.  I don't know if it is still that
> way, but it seems like reasonable solution to me.

Actually in the past year many of the gaming studios have
started to do this for the release process.  It will start during an
off-peak hour and then flow through the day.  The end-user induced
demand and stress it places on networks doesn't fit into the historical
95/5 30-day peak planning model.  I know some carriers are struggling to
adjust.  There's a few of us here on-list, please reach out so we can
help.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: wow, lots of akamai

2021-04-02 Thread Jared Mauch
On Thu, Apr 01, 2021 at 08:09:24PM +, Luke Guillory wrote:
> IX’s don’t really help the source doesn’t use them.
> 
> Akamai traffic.
> 17G via Local Cache
> 17G via Transit
> 8G via IXs.
> 
> Plenty of room on IXs for more on our side.

Often we can see the ports at the IX flatline in these cases.
I've been working to improve some of them, for exampe at the DetroitIX
we have 300G of capacity and you can see the impact at the IX here:

https://www.detroitix.com/#stats

If you look at the monthly graph you can see other game download events.

It can be incredibly hard to right-size this stuff, please do reach out
to myself or Niels if things didn't work right for your network.  Also
you should reach out to your carriers who may have had congestion to
ensure they are upgrading their networks as well.  I've been told by
some teams they won't upgraded for a day a month of thsi big traffic,
but when one of their customers has issues they immediately escalate to
us to ask us to move (and we try hard to do that).

It's not all throw bandwidth at the problem, but with 100g optics being
so cheap these days, it may be the lower bar

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: Google IP Geolocation

2021-04-10 Thread Jared Mauch
I've had a similar issue in the past trying to get ready to peer with them. I 
wanted portal access to look at things. I may yet post a geofeed file just 
because. 

(I was also rejected a portal account, didn't escalate to friends at google 
because I know my traffic is under a gig for now). 

- Jared 
 
Ps: Akamai can take geofeed if you have issues

Sent from my TI-99/4a

> On Apr 10, 2021, at 4:57 AM, Laura Smith via NANOG  wrote:
> 
> Yup. I've had this problem with Google for two years now.
> 
> "We're Google. We know better than you. We're not interested in discussion. 
> And no, you can't have access to the ISP portal you silly little person"  
> .. is the summary of my experience.
> 
> And all this is despite my network peering with Google over a major IXP.  
> They *STILL* can't get the right geolocation despite having a direct peering 
> session with us over the exchange !
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐ Original Message ‐‐‐
>> On Monday, 29 March 2021 21:12, Troy Kelly via NANOG  wrote:
>> 
>> We've also been denied access to the ISP portal.
>> 
>> When we replied as to why, we were told to not open another ticket. They 
>> aren't interested in conversation.
>> 
>> Sent from ProtonMail mobile
>> 
>>  Original Message 
>>> On 30 Mar 2021, 6:53 am, Mike Hammett < na...@ics-il.net> wrote:
>>> 
>>> I've had others at Google specifically say that portal should be used for 
>>> that purpose, so maybe they need to make sure right and left hands know 
>>> what the other is doing.
>>> 
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>> 
>>> From: "Josh Luthman" 
>>> To: "Christopher Morrow" 
>>> Cc: nanog@nanog.org
>>> Sent: Monday, March 29, 2021 1:52:48 PM
>>> Subject: Re: Google IP Geolocation
>>> 
>>> https://isp.google.com
>>> 
>>> I signed up for an account there and they said:
>>> 
>>> "Currently, the Google ISP portal is designed for our partners of GGC, PNI 
>>> or IX programs.
>>>  
>>> The access to portal is granted on request only to our partners." 
>>> 
>>> Josh Luthman
>>> 24/7 Help Desk: 937-552-2340
>>> Direct: 937-552-2343
>>> 1100 Wayne St
>>> Suite 1337
>>> Troy, OH 45373
>>> 
 On Mon, Mar 29, 2021 at 2:08 PM Christopher Morrow 
  wrote:
>>> 
 On Mon, Mar 29, 2021 at 1:59 PM Josh Luthman  
 wrote:
 
> Google ISP specifically told me they didn't want to do deal with 
> geolocation on the ISP portal.
 
 unsure who 'google isp' is here, but ... I think if you point at a 
 properly formed geo-location data file it'll get eaten and produce proper 
 results for you.
 you do have to add that in your isp portal:
tri-pipe-thingy -> configuration -> IP geolocation
 and /register feed/ button on that page.
  
 
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Sat, Mar 27, 2021 at 3:28 PM Christopher Morrow 
>  wrote:
> 
>> As a note, on the ISP portal there's a place to put a link to your 
>> RFC8805 format geolocation feed...
>> these are scraped out 'regularly' and help keep things oriented better 
>> for folks.
>> 
>> the ietf noc folk use this method to tell google (and other folk who 
>> scrape out our data) where meetings are before the meetings get there:
>>   https://noc.ietf.org/geo/google.csv
>> 
>> -chris
>> volunteer noc persona
>> 
>> On Fri, Mar 26, 2021 at 6:43 PM Michael K. Spears  
>> wrote:
>> 
>>> Awesome, I think I’ve figured out the Google ISP portal signup, but it 
>>> definitely seems semi-complicated in a way, notably finding the link…
>>> 
>>>  
>>> 
>>> Thank you,
>>> 
>>> Michael K. Spears
>>> 
>>> 727.656.3347
>>> 
>>>  
>>> 
>>> From: Mike Hammett 
>>> Sent: Friday, March 26, 2021 6:30 PM
>>> To: Michael K. Spears 
>>> Cc: nanog@nanog.org
>>> Subject: Re: Google IP Geolocation
>>> 
>>>  
>>> 
>>> We're working on a video to show people how to sign up for the ISP 
>>> portal and get to that part of the portal once signed up. We'll drop a 
>>> link to it near the Google section of our geolocation page.
>>> 
>>> -
>>> 
>>> Mike Hammett
>>> 
>>> Intelligent Computing Solutions
>>> 
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> 
>>> http://www.midwest-ix.com
>>> 
>>> From: "Michael K. Spears" 
>>> 
>>> To: nanog@nanog.org
>>> 
>>> Sent: Friday, March 26, 2021 5:10:23 PM
>>> 
>>> Subject: Google IP Geolocation
>>> 
>>> Anyone have a good contact at Google who can help with IP geolocation? 
>>> I have a /24 where anything related to Google is in the wrong language.
>>> 
>>> Thank you,
>>> 
>>

Re: Muni broadband sucks (was: New minimum speed for US broadband connections)

2021-06-02 Thread Jared Mauch



> On Jun 2, 2021, at 12:44 PM, Andy Ringsmuth  wrote:
> 
>>> On Mon, May 31, 2021 Mike Hammett wrote:
 Muni broadband does suck, but that's another thread for another day.
>>> Excluding cases where muni broadband doesn't suck, why does muni broadband 
>>> suck?
>>> 
>>> Personally I wouldn't mind more access to dark fiber à la Stokab, much like 
>>> the dry copper pairs of yesterday.
>>> 
>>> If the default state of muni broadband of is suck, what is the root cause? 
>>> Is it a people problem and/or can something be done to improve on the 
>>> default state?
>> 
>> 
>> Muni broadband sucks for several reasons but the most important one is:
>> 
>> Competition. Municipal broadband eliminates it. If it's not obvious
>> why, feel free to Google how competition and monopolization impact
>> product quality. It's a pretty universal trait.
>> 
>> 
>> If you were to structure muni broadband to enhance competition rather
>> than limit it, you might get a different result. For example, if
>> municipalities installed and leased fiber optic cables to every
>> structure but didn't provide any services on those cables, relying
>> instead on third parties directly billing the customer to do so, it
>> could work out as well as having municipalities pay for roads and
>> letting people buy their own cars and trucks to use on them.
> 
> In many municipalities, you can choose your electricity provider. And yet 
> there are not multiple companies running power lines to every house.
> 
> It is easy to make the argument “muni broadband sucks because no competition” 
> but it is much more difficult to back it up with hard data.
> 
> Take a look at Nebraska for instance. Here, by law, electricity is a public 
> utility. And yet we have some of the lowest rates and highest uptime in the 
> country. No competition, low prices, stellar service record.
> 
> I’m generally all for private enterprise. But when those private enterprises 
> take public money, don’t do what they are supposed to do with it, squander 
> it, and nothing changes, again and again, well, what’s that definition of 
> insanity?
> 
> 
> Here in Lincoln, Nebraska, we actually do have fiber available at every 
> address in the city. And a private company did it. 100 percent underground, 
> all 96 square miles of the city. They did it all in about two years. I can 
> get 50Mbps synchronous for $45, 500 for $70 or gig for $99. TV and phone also 
> if I want it. Local support too, not India. 
> 
> They now have fiber in 15 Nebraska cities and two in Colorado and are 
> expanding rapidly. Why? A great product at a great price with outstanding 
> customer service. Spectrum is losing customers like crazy as a result, and 
> precisely zero people are shedding any tears (Spectrum salesmen excepted).
> 
> It can be done. Is it an investment? Yes. Just like anything else. Some 
> investments have a quicker return on capital than others.

+1 on the investment lifecycle requirement.  It can require a 10-20 year 
vision.  The problem we have right now is due to squirrel chasing on the part 
of some companies the money that could have been invested in locking in markets 
was spent on other investments.  You see a big difference between forward 
looking companies and their network performance and those that are backwards 
looking.

I had to build fiber to my house because the fiber near my home (about 1200’ 
away) was not in a position to be upgraded or maintained in such a way to 
deliver service to our area.  This is a very common trend I’ve observed.

My county did a large broadband survey where a contractor drove by every 
home/property to determine what was there.  Many addresses without service have 
multiple fiber providers at the road, it’s just not the “right fiber”.  This 
also includes spare conduit and space that was built out in a forward thinking 
model that others have to duplicate later because the assets are lost or 
forgotten in paperwork.

I also see a number of the smaller ISPs (and some bigger ones) who are like 
“you can watch Netflix and zoom, what’s your problem?” When there are end-users 
that are willing to pay for the higher speeds.  Not every home is going to 
spend $8k or $100k to get service, but they certainly do exist and make the 
business case more feasible.

- Jared

Internet Quality workshop CFP for the internet architecture board (IAB)

2021-07-06 Thread Jared Mauch
I know that many operators shift traffic based on network and internet quality 
(or don’t use certain networks at all).  This is a great chance to share 
information about things operators have experienced or do to actively measure 
or otherwise to inform those decisions.

- snip -

For complete details, please see:
https://www.iab.org/activities/workshops/network-quality/

Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
Invitations Issued by: Monday 16th August 2021

Workshop Date: This will be a virtual workshop, spread over three days:

1400-1800 UTC Tue 14th September 2021
1400-1800 UTC Wed 15th September 2021
1400-1800 UTC Thu 16th September 2021

Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira

The Program Committee members:

Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.

Send Submissions to: network-quality-workshop...@iab.org.

Position papers from academia, industry, the open source community and
others that focus on measurements, experiences, observations and
advice for the future are welcome. Papers that reflect experience
based on deployed services are especially welcome. The organizers
understand that specific actions taken by operators are unlikely to be
discussed in detail, so papers discussing general categories of
actions and issues without naming specific technologies, products, or
other players in the ecosystem are expected. Papers should not focus
on specific protocol solutions.

The workshop will be by invitation only. Those wishing to attend
should submit a position paper to the address above; it may take the
form of an Internet-Draft.

All inputs submitted and considered relevant will be published on the
workshop website. The organisers will decide whom to invite based on
the submissions received. Sessions will be organized according to
content, and not every accepted submission or invited attendee will
have an opportunity to present as the intent is to foster discussion
and not simply to have a sequence of presentations.

Position papers from those not planning to attend the virtual sessions
themselves are also encouraged. A workshop report will be published
afterwards.

Overview:

"We believe that one of the major factors behind this lack of progress
is the popular perception that throughput is the often sole measure of
the quality of Internet connectivity. With such narrow focus, people
don’t consider questions such as:

What is the latency under typical working conditions?
How reliable is the connectivity across longer time periods?
Does the network allow the use of a broad range of protocols?
What services can be run by clients of the network?
What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
What security mechanisms are available for local services, such as DNS?
To what degree are the privacy, confidentiality, integrity and
authenticity of user communications guarded?

Improving these aspects of network quality will likely depend on
measurement and exposing metrics to all involved parties, including to
end users in a meaningful way. Such measurements and exposure of the
right metrics will allow service providers and network operators to
focus on the aspects that impacts the users’ experience most and at
the same time empowers users to choose the Internet service that will
give them the best experience."


Re: A crazy idea

2021-07-19 Thread Jared Mauch



> On Jul 19, 2021, at 9:04 AM, Stephen Satchell  wrote:
> 
> On 7/19/21 5:41 AM, Feldman, Mark wrote:
>> What you propose is not outlandish; some ISPs have been dual stack
>> and providing some combination of these services for years.  They
>> already provide IPv6 ip6.arpa delegations should their business
>> customers want them.  Some even provide at least a /56 so customers
>> can have multiple /64 subnets.  And we, I mean they, can also provide
>> RFC2317 in-addr.arpa delegations for those smaller IPv4 blocks.
> 
> The part that is missing isn't the "some ISPs", it's "all ISPs".  Also, I 
> don't know of any DNS service provider that offers a product to handle 
> delegations from the IN-ADDR.ARPA and IP6.ARPA trees.
> 
> I'm focusing on the SOHO customer market with my proposal.

Most are regional carriers w/ monopoly and no incentive to offer anything else. 
 This is especially the case with AT&T in my area.  The same applies to other 
regional providers like Frontier and even services like Verizon FIOS that do 
not offer IPv6 at all.

You really want a SMB connection w/ dedicated IP space, and they may not be 
able to sell that to you on their consumer network.

- Jared

Re: 100G, input errors and/or transceiver issues

2021-07-19 Thread Jared Mauch



> On Jul 19, 2021, at 1:50 PM, Saku Ytti  wrote:
> 
> On Mon, 19 Jul 2021 at 20:19, Graham Johnston
>  wrote:
> 
>> I don't at this point have long term data collection compiled for the issues 
>> that we've faced. That said, we have two 100G transport links that have a 
>> regular background level of input errors at ranges that hover between 
>> 0.00055 to 0.00383 PPS on one link, and none to 0.00135 PPS (that jumped to 
>> 0.03943 PPS over the weekend). The range is often directionally associated 
>> rather than variable
> 
> On typical 100G link we do not get single FCS error in a typical day.
> However Ethernet spec still allows very high error rate of 10**-12. So
> 1 error per 1Tb (b not B). I.e. 1 error per 10s, or 0.1PPS would be
> in-spec. We see much better performance to this and vendors generally
> accept lower error rates as legitimate errors.
> 

I will confirm my experience with this at $dayjob as well.  We see interfaces 
with no errors over much longer periods of time inclusive of several of the 
technology options.  If you are seeing errors, there’s likely something wrong 
like unclean fiber or bad optic/linecard etc.

- Jared



Re: Alien waves

2021-07-21 Thread Jared Mauch
I looked at this before and go far enough in the conversations with one carrier 
that had sold this as a product before and had a poor experience with the 
customers they were no longer offering it. You are likely better off getting a 
volume deal on waves, which can be had for pretty cheap these days.

I can get 400g waves from carriers now, the capacity involved is short of what 
makes sense for IRU long haul.

Dark in the metro continues to work out. As Saku mentioned there's significant 
risks opening up a system even with channel blockers etc to protect each other 
it hasn't worked out. 

Ask some carriers about a dedicated managed system in your rack if XC fees are 
killing you. 

Sent from my TI-99/4a

> On Jul 20, 2021, at 4:35 PM, Lady Benjamin Cannon of Glencoe, ASCE 
>  wrote:
> 
> Does anyone have a comprehensive (or any) list of carriers doing alien 
> wavelengths? (background: 
> https://thecinict.com/2021/03/05/adding-alien-wavelengths/ 
> https://www.ekinops.com/solutions/optical-transport/alien-wavelength )
> 
> Emphasis on subsea operators.
> —L.B.
> 
> Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
> 6x7 Networks & 6x7 Telecom, LLC 
> CEO 
> l...@6by7.net
> "The only fully end-to-end encrypted global telecommunications company in the 
> world.”
> FCC License KJ6FJJ
> 
> 
> 


Re: Global Akamai Outage

2021-07-22 Thread Jared Mauch



> On Jul 22, 2021, at 12:56 PM, Andy Ringsmuth  wrote:
> 
> The outage appears to have, ironically, taken out the outages and 
> outages-discussion lists too.
> 
> Kinda like having a fire at the 911 dispatch center…



Should not have impacted me in my hosting of the list.  Obviously if the domain 
names were impacted in the lookups for sending e-mail as well, there would be 
problems.



- Jared





Re: Global Akamai Outage

2021-07-25 Thread Jared Mauch
Work hat is not on, but context is included from prior workplaces etc. 

> On Jul 25, 2021, at 2:22 AM, Saku Ytti  wrote:
> 
> It doesn't seem like a tenable solution, when the solution is 'do
> better', since I'm sure whoever did those checks did their best in the
> first place. So we must assume we have some fundamental limits what
> 'do better' can achieve, we have to assume we have similar level of
> outage potential in all work we've produced and continue to produce
> for which we exert very little control over.

I have seen a very strong culture around risk and risk avoidance whenever 
possible at akamai. Some minor changes are taken very seriously. 

I appreciate that on a daily basis, and when we make mistakes (I am human after 
all) are made, reviews of the mistakes and corrective steps are planned and 
followed up on. I'm sure this time will not be different. 

I also get how easy it is to be cynical about these issues. There's always 
someone with power who can break things, but those can also often fix them just 
as fast. 

Focus on how you can do a transactional routing change and roll it back, how 
you can test etc. 

This is why for years I told one vendor that had a line-by-line parser their 
system was too unsafe for operation. 

There's also other questions like:

How can we improve response times when things are routed poorly? Time to 
mitigate hijacks is improved my majority of providers doing RPKI OV, but 
interprovider response time scales are much longer. I also think about the two 
big CTL long haul and routing issues last year. How can you mitigate these 
externalities. 

- Jared 

AS20940 Max Prefix and as-path filtering

2021-08-18 Thread Jared Mauch
Hello,

Akamai has been increasing the routes we are advertising in various locations 
as part of ongoing network changes.  If you have a max-prefix set for Akamai 
can you please increase v4 to 1k and ensure you are accepting our additional 
ASNs that may live behind the clusters.  

We are going to be updating PeeringDB to reflect this change as well.

An appropriate as-path access-list would look like:

20940_(21342|16625|33905|36183|24319|34164|35204|35994|43639|18680|39836|35993|23454|18717|31108|36183)

Feel free to reach out if you have any questions.

- Jared

Re: Setting sensible max-prefix limits

2021-08-18 Thread Jared Mauch



> On Aug 18, 2021, at 9:38 AM, John Kristoff  wrote:
> 
> Maybe because there isn't a simple, universal approach to setting it.
> Probably like a lot of people, historically I'd set it to
> some % over the current stable count and then manually adjust when the
> limits were about to be breached, or often was the case when they were
> and I wasn't ready for it. Not ideal.
> 
> I've never felt the automation of this setting however was worth the
> effort.  Of course I am not usually responsible for hundreds of routers
> and thousands of peering sessions.

We did a variant of this at NTT, with certain baseline settings.  Sometimes 
networks would advertise more routes because they onboarded a large customer 
and it would cause manual updates to be necessary.

Polling daily and snapshotting these values is important to understand what is 
changing.  The reason I just posted a message about Akamai max-prefix is we 
have been giving some general guidance that is out of line/norm compared to 
what perhaps what we want.  This won’t cause a service outage per-se but will 
cause suboptimal routing as we continue to make improvements and upgrades to 
our network.

- Jared

Re: Reminder: Never connect a generator to home wiring without transfer switch

2021-08-25 Thread Jared Mauch



> On Aug 25, 2021, at 10:04 AM, Mark Tinka  wrote:
> 
> You need to make these things fool-proof. We haven't traveled in over a year 
> but the day we do, it's a recipe for disaster if the person that deals with 
> this stuff is on the road when the power goes out back at home.

This is why I personally spent the $$ on a proper standby generator with 
multiple ATS for the multiple panels.

- Jared

Re: FYI - Suspension of Cogent access to ARIN Whois

2020-01-07 Thread Jared Mauch
I have to +1 this. I've been solicited many times by them myself and it's sad 
to see the information used that way. 

When I worked at another carrier I helped stop this as well with the sales 
people. They were creative, but it does at least violate the social norms of 
the industry at minimum and that was enough for me. 

I've been sad to see all the valuable contact methods disappear as the industry 
has grown over the past 25 years 

Sent from my iCar

> On Jan 7, 2020, at 5:49 PM, Matt Harris  wrote:
> 
> 
>> On Tue, Jan 7, 2020 at 4:46 PM Martin Hannigan  wrote:
> 
>> 
>> 
>>> On Tue, Jan 7, 2020 at 08:51 John Curran  wrote:
>>> On 7 Jan 2020, at 5:01 AM, Martijn Schmidt via NANOG  
>>> wrote:
>>> > 
>>> > Out of curiosity, since we aren't affected by this ourselves, I know of 
>>> > cases where Cogent has sub-allocated IP space to its customers but which 
>>> > those customers originate from their own ASN and then announce to 
>>> > multiple upstream providers.
>>> > 
>>> > So while the IP space is registered to Cogent and allocated to its 
>>> > customer, the AS-path might be something like ^174_456$ but it's entirely 
>>> > possible that ARIN would observe it as ^123_456$ instead. Are such IP 
>>> > address blocks affected by the suspension?
>>> 
>>> As noted earlier, ARIN has suspended service for all Cogent-registered IP 
>>> address blocks - this is being done as a discrete IP block access list 
>>> applied to relevant ARIN Whois services, so the routing of the blocks are 
>>> immaterial - a customer using a suballocation of Cogent space could be 
>>> affected but customers with their own IP blocks blocks that are simply 
>>> being routed by Cogent are not affected. 
>> 
>> 
>> 
>> 
>> This is a disproportionate response IMHO. $0.02
>> 
>> YMMV,
>> 
>> -M<
> 
> Seems entirely reasonable to me. You break the rules, you lose the privilege. 
> Works the same way with my 7 year old. 
> 


Re: IPv6 Prefix Delegation to customers.

2020-01-16 Thread Jared Mauch
Arista/Cisco have commands like this:

ipv6 dhcp relay install routes

You place on the interface to make this happen.

- Jared


> On Jan 16, 2020, at 11:27 AM, Chris Gross  wrote:
> 
> In my environment I’ve been running Kea dhcp6 against Ciscos of varying 
> platform (7600, ASR920, etc) and just them as a relay. In this case, the 
> Cisco itself is installing a route as it snoops the relay action 
> automatically. This was one of the harder things to wrap my head around 
> before just slapping it in to see what happened and bam, routes. Router gets 
> a WAN IP from the loopback via DHCPv6 as well, then gets PD assigned after.
> 
> interface Loopback10
> vrf forwarding CGNAT
> no ip address
> ipv6 address 2001:DB8::1/64
> !
> interface Vlan
> vrf forwarding CGNAT
> ip address 100.64.Y.Z 255.255.252.0
> ip helper-address global 10.0.Y.Z
> ip helper-address global 10.0.Y.Z
> ip flow ingress
> load-interval 30
> ipv6 address FE80::1 link-local
> ipv6 enable
> ipv6 nd router-preference High
> ipv6 dhcp relay destination 2001:DB8:0:A::BEEF source-address 2001:DB8:YZ01::1
> ipv6 dhcp relay destination 2001:DB8:0:B::BEEF source-address 2001:DB8:YZ01::1
> 
> S   2001:DB8:YZ00:3F00::/56 [1/0]
>  via FE80::4665:7FFF:FE14:EDC2, Vlan
>  
> Chris Gross
> Network Architect
>  
> From: NANOG  On Behalf Of Brandon Price
> Sent: Wednesday, January 15, 2020 9:01 PM
> To: nanog list 
> Subject: IPv6 Prefix Delegation to customers.
>  
> CAUTION: This email originated from outside NineStar Connect. Do not click 
> links or open attachments unless you recognize the sender and know that the 
> content is safe. If you have any concerns, click here to open a ticket with 
> the NetAdmin team.
>  
>  
> Hey Nanog,
>  
> I am in the process of building out a FTTH proof of concept, and I would 
> really like to offer each of my customers a /48 of IPv6.
> I’ve been able to announce my /32 to my upstreams, dual-stack all of my 
> internal infrastructure no-problem, build v6 recursive name servers, etc.
> This was fairly straight-forward.
>  
> Where I am struggling is the Prefix Delegation part. How are most folks 
> getting the PD subnets into their IGPs? In my environment I don’t run the 
> DHCP server process on the router that is directly connected to the clients. 
> I have seen documentation that cisco and juniper DHCPv6 processes are smart 
> enough to insert that prefix into the routing table when they hand it out, 
> but how is this handled in an environment with a central DHCP server? I do 
> not currently run any PPPOE in my environment and I don’t use RADIUS for the 
> subscriber management. I would really just like to stick to DHCP ideally.
>  
> If anyone has any pointers, I would appreciate it.
>  
> Brandon Price
> Senior Network Engineer
> City of Sherwood, Sherwood Broadband
> Desk: 503.625.4258
> Cell: 971.979.2182
>  
> This email may contain confidential information or privileged material and is 
> intended for use solely by the above referenced recipient. Any review, 
> copying, printing, disclosure, distribution, or other use by any other person 
> or entity is strictly prohibited and may be illegal. If you are not the named 
> recipient, or believe you have received this email in error, please 
> immediately notify the City of Sherwood at (503) 625-5522 and delete the copy 
> you received.



Re: akamai yesterday - what in the world was that

2020-01-23 Thread Jared Mauch



> On Jan 23, 2020, at 10:16 AM, Kaiser, Erich  wrote:
> 
> Yeah we saw that as well. Must be a game release or something.

Yes, that’s my understanding as well.

- Jared

Re: akamai yesterday - what in the world was that

2020-01-23 Thread Jared Mauch



> On Jan 23, 2020, at 11:52 AM, Valdis Klētnieks  
> wrote:
> 
> On Thu, 23 Jan 2020 17:13:15 +0100, Bryan Holloway said:
> 
>> Game releases are hardly a new thing, but these last two events seem to
>> be almost an order of magnitude higher than what we're used to (at least
>> on our predominantly eyeball network.)
>> 
>> Any thoughts from the community? We're taking steps to accommodate, but
>> from a capacity-planning perspective, this seems non-linear to me.
> 
> Be prepared for an entire new world of hurt this holiday season. Sony has 
> already
> confirmed that PS5 releases will ship on 100Gbyte blu-ray disks.  Which means 
> that
> download sizes will be comparable…

There’s also the “we will stream you all the data things” I keep hearing about 
like the
Consoles without discs or some other thing I can’t remember the name of.

I think this is a tribute to how we’ve built and upgraded networks for capacity 
and speed.

- Jared



Re: akamai yesterday - what in the world was that

2020-01-23 Thread Jared Mauch



> On Jan 23, 2020, at 1:25 PM, Warren Kumari  wrote:
> 
> On Thu, Jan 23, 2020 at 12:45 PM Hugo Slabbert  wrote:
>> 
>>> This just follows the same rules as networks have always seemed to; If you 
>>> build it, they will come, and you'll have to build more. :)
>> 
>> https://en.m.wikipedia.org/wiki/Induced_demand
>> 
> 
> Yup, there is also (in networking at least) suppressed demand -- I'm
> sure we've all seen capacity planning discussions along the lines of:
> "My 1GE is running at 95% capacity - I'm replacing it with a 10GE and
> it will be around 10% used... wait?! What?! It's now at 7Gbps?! How
> the hell did that happen?!" scenarios. They usedto be funny, but these
> days I just find it depressing...

I find it both happy and disturbing.  I remember the first 2.4/2.5g links I 
turned up as well as the first 10g and (eventually) the first 100g links.

I was leaving the house earlier this week thinking about how it used to be Mbps 
of traffic that was a lot and now it’s Gbps and how that’s shifted to Tbps.

While it makes me feel old, it’s also something that I marvel about 
periodically.

- jared

Re: akamai yesterday - what in the world was that

2020-01-23 Thread Jared Mauch



> On Jan 23, 2020, at 2:21 PM, Kaiser, Erich  wrote:
> 
> Except the CDN providing the content did not anticipate this type of influx 
> (How come I am not sure probably more concerned about new business revenue 
> and not thinking about the backend infrastructure) and has pushed over costly 
> peers for most of us.  BTW, we are still waiting for our PNIs with them.
> Very frustrating…

Can be for large orgs that are dysfunctional as well.  This is somewhat typical 
of large companies and we usually try to not air all our internal drama in 
public.

It’s entirely possible there’s numerous problems with blame to go around.

It can also take longer than we’d like to get things done as well.

I can assure you everyone is acting in good faith here, even if it doesn’t seem 
that way.  Sometimes errors aren’t caught immediately and sometimes people go 
on leave, etc..  The rest of the dialogue should happen off-list though.

- Jared



Re: akamai yesterday - what in the world was that

2020-01-23 Thread Jared Mauch
Apple did this with the original iPhone. Turned out even in their ecosystem 
they didn't get it right. The full restore images have always been there and 
diffs didn't reappear until you could "OTA" the device (WiFi)

I can't imagine how hard a console would be with every random app writing data 
wherever. 

Sandboxes and jails have been escaped as long as they have been around as well 
so they can help but are far from perfect 

Sent from my iCar

> On Jan 23, 2020, at 6:21 PM, Mike Hammett  wrote:
> 
> 
> If true (not arguing), that's really dumb.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> 
> Midwest-IX
> http://www.midwest-ix.com
> 
> From: "Brandon Martin" 
> To: nanog@nanog.org
> Sent: Thursday, January 23, 2020 10:23:24 AM
> Subject: Re: akamai yesterday - what in the world was that
> 
> On 1/23/20 11:13 AM, Bryan Holloway wrote:
> > This echoed events a month or so ago, and I'm curious as to what is 
> > making these releases more, uh, network-impacting.
> 
> My understanding is that, in addition to factors others have mentioned 
> (games are larger, more network based delivery, etc.), that there's a 
> move AWAY from differential patching, to the extent it was previously 
> being used, toward simply delivering an entire new copy of the game, 
> including assets that completely duplicate those that someone may 
> already have.
> 
> Apparently the rationale is that this is easier on the publisher and 
> those preparing the release, which allows them to get things out sooner, 
> since they don't have to come up with a decent differential patcher and 
> can just make use of the delivery mechanisms already present on the 
> content platform the user is already using.
> 
> When you've got 100GB games with huge market penetration and each 
> "patch" is an entirely new copy of said 100GB game, that's a lot of traffic.
> -- 
> Brandon Martin
> 


Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC

2020-01-27 Thread Jared Mauch
You will still see backscatter which will let you know your space is being 
spoofed as well. 

This will show in your telemetry. 

Sent from my iCar

> On Jan 27, 2020, at 7:57 PM, Mike Hammett  wrote:
> 
> 
> How would they know what to look for?
> 
> I'm assuming Sony isn't cooperating.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> 
> Midwest-IX
> http://www.midwest-ix.com
> 
> From: "Ben Cannon" 
> To: "Mike Hammett" 
> Cc: "Roland Dobbins" , "NANOG Operators' Group" 
> 
> Sent: Monday, January 27, 2020 6:40:25 PM
> Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
> 
> Transit carriers could work the flows backwards.
> 
> -Ben Cannon
> CEO 6x7 Networks & 6x7 Telecom, LLC 
> b...@6by7.net
> 
> 
> 
> 
> On Jan 27, 2020, at 4:39 PM, Mike Hammett  wrote:
> 
> If someone is being spoofed, they aren't receiving the spoofed packets. How 
> are they supposed to collect anything on the attack?
> 
> Offending host pretending to be Octolus -> Sony -> Real Octolus.
> 
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> 
> Midwest-IX
> http://www.midwest-ix.com
> 
> From: "Roland Dobbins" 
> To: "Octolus Development" 
> Cc: "Heather Schiller via NANOG" 
> Sent: Monday, January 27, 2020 6:29:16 PM
> Subject: Re: Reaching out to Sony NOC, resolving DDoS Issues - Need POC
> 
> 
> 
> On Jan 28, 2020, at 04:12, Octolus Development  wrote:
> 
> It is impossible to find the true origin of where the spoofed attacks are 
> coming from.
> 
> This is demonstrably untrue. 
> 
> If you provide the requisite information to operators, they can look through 
> their flow telemetry collection/analysis systems in order to determine 
> whether the spoofed traffic traversed their network; if it did so, they will 
> see where it ingressed their network. 
> 
> With enough participants who have this capability, it's possible to trace the 
> spoofed traffic back to its origin network, or at least some network or 
> networks topologically proximate to the origin network. 
> 
> That's what Damian is suggesting. 
> 
> 
> Roland Dobbins 
> 
> 


Re: akamai yesterday - what in the world was that

2020-02-11 Thread Jared Mauch
Looking good from my perspective. Let me know if we are causing you pain and 
let's see what can be done to improve. 

I'm here in SF if you are at nanog. 

Sent from my iCar

> On Feb 11, 2020, at 3:42 PM, Tom Deligiannis  
> wrote:
> 
> There is a major update that has released today, how's everything looking for 
> everyone?


Re: akamai yesterday - what in the world was that

2020-02-12 Thread Jared Mauch
Our CEO tweeted out a new platform peak yesterday which when you round it from 
the many trailing 9’s is 140T.

https://twitter.com/TomLeightonAKAM/status/1227389107665592320

I want to ensure the bits get through with the least pain as possible.

I’m expecting more of the same in the future, so if you’re not talking to 
someone here and we caused you some pain, I can help.  A number of people 
reached out in the past 24h and I think i’m all caught up on those, but while 
regular IXP peering can help in many places, if we can offload on a high 
capacity PNI that is my preference.  

- Jared

> On Feb 12, 2020, at 10:33 AM, Jason Canady  wrote:
> 
> We saw a higher load overnight, a little bit of a spike last night, but 
> really hard to tell overall with our traffic.  Updates were still going at 
> 8am today.  We run a local/regional WISP.
> 
> On 2/12/20 9:46 AM, Brandon Martin wrote:
>> On 2/11/20 6:41 PM, Tom Deligiannis wrote:
>>> There is a major update that has released today, how's everything looking 
>>> for everyone?
>> 
>> I run a couple distinct very small networks.  Both are transit-only with no 
>> direct peering or local caching and generally sub-gbps.
>> 
>> One set a new 1-min 95% record and did so by nearly 25% over its previous 
>> record.  The other matched its existing record.  The former thankfully has 
>> ample capacity, while the latter thankfully did so before primetime.
>> 
>> These are definitely going to be harder to manage than the primetime hump.  
>> It would be nice if things could drop overnight to hopefully spread things 
>> out during the daytime lull some.  I understand that some people won't pick 
>> it up until they get home and turn on devices (which is often after school 
>> hours for these type of game updates), but spreading things out would be 
>> really nice especially for those of us without local caching.
>> 



Re: akamai yesterday - what in the world was that

2020-02-12 Thread Jared Mauch
When you see this please raise it to my attention. I can't promise a resolution 
but will promise clarity in what is going on. 

I know some cities are problematic as we are moving cages or datacenter space 
and have the usual related problems. 

There is always something. 

Sent from my iCar

> On Feb 12, 2020, at 9:36 AM, Seth Mattinen  wrote:
> 
> 
> The wheels of bureaucracy are certainly a problem. The largest peer on our 
> local exchange couldn't even get Akamai to complete a peering turn up because 
> whoever was working on the ticket on the Akamai side got stuck on trying to 
> set up the wrong location. And then months pass, it never got resolved, and 
> then they decided to pull the cache. Akamai had one hand failing to set up 
> new peers and the other hand saying why aren't there more peers, and the two 
> hands never know what the other is doing.


Re: ATT Microcell in Austin, TX

2020-02-18 Thread Jared Mauch
On Tue, Feb 18, 2020 at 02:05:15PM -0500, Shane Ronan wrote:
> I can tell you that most carriers have neither type, at least in the US.

Most towers can survive a brown/blackout lasting a few minutes
(usually enough to last the safety system cycling and the fuses blowing
on a grounded leg).

Major towers tend to have 8-12h of battery, with sometimes 24h.

Most utility outages are restored in that time with the exception
being major storms.

In michigan we can get a credit for lack of restoration in 16 hours:

-- snip --
A customer is eligible for a credit under normal
conditions if the utility fails to restore service
within 16 hours after an outage resulting from
conditions other than catastrophic conditions.
-- snip --

this lines up with the planning strategy of the utilities.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: QUIC traffic throttled on AT&T residential

2020-02-18 Thread Jared Mauch
The members of the QUIC WG at IETF have not thought this was a problem as they 
don't observe it across the board. 

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - 
this is also not considered an issue either. 

The explanation I got (which seems fair) from someone was that they only way to 
roll a new transport was to squat on some existing stuff that would make it 
through firewalls. 

We have an internet that is largely fixed around running NATs and TCP and UDP 
only these days. I for one find this sad and don't see a good way forward on 
either side. 

Sent from my iCar

> On Feb 18, 2020, at 7:13 PM, Daniel Sterling  
> wrote:
> 
> 9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar  wrote:
>> 
>> Are you suggesting that ATT block all QUIC across their network?
> 
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT&T ipv4 residential connection (and a web search
> suggests I'm not alone).


Re: QUIC traffic throttled on AT&T residential

2020-02-20 Thread Jared Mauch
On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
> On 2/20/2020 10:34 AM, Ca By wrote:
> > 
> > On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson  > <mailto:bl...@ispn.net>> wrote:
> > Dropping udp is not from a “best practice” doc from a vendor, it is
> > deployed by network ops folks that are trying to sleep at night.
> 
> I get it Ca, I happen to be one of those network ops folks that likes to
> sleep at night. However, I've never thought it was a good practice to break
> applications in fun ways for my customers to discover on their own and I've
> never sold someone a 150Mbps package that actually only delivers 10Mbps for
> certain applications. Regardless of the intent, ATT and Cox's policies are
> not transparent, open, or neutral on this topic. This leaves us to speculate
> on what their intentions might have been and whether their actions are an
> appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as
far as the UDP/1434 filters I was forced to deploy due to persistent network
congestion.  Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, 
chargen
and other activities.

Extending these to consumer side where the traffic often originated 
makes
sense until the devices can be secured.  You can blame the providers for 
deploying
filters, or not disconnecting consumers that have devices that can be exploited
or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: QUIC traffic throttled on AT&T residential

2020-02-20 Thread Jared Mauch
On Thu, Feb 20, 2020 at 02:50:58PM -0500, Todd Underwood wrote:
> and just to check one thing...
> 
> On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
> wrote:
> 
> > I don't particularly *want* to block or advocate blocking QUIC, but if
> > I keep hitting the issue and can't help people troubleshoot, what
> > other sane option have I?
> >
> 
> i don't think you've addressed the "replace your broken ISP" action that is
> clearly sane and would fix this, right?
> 
> i'm assuming that this is not an option to get a functional IP layer?

often one has to live in the world of reality.  if google is very worried about
the broadband experience they can continue to invest in this space and 
challenges.

it seems the corporate overlords has led it to not continue to persue this 
option.

it's hard to have it both ways, wanting unlimited UDP, low cost (free?) ports 
for
bits, etc..

there's natural dynamics at play here with business interests that haven't quite
balanced out yet.  perhaps the LTE/5G overlay will replace fixed broadband at 
home
with managed CPE from the cellular providers.

if the question is will the browser vendor (google) or the broadband provider 
(att)
move first, i can already predict the answer.  my experience (again) with the 
quic
wg is they seem to think there's many options and bad providers will be replaced
which seems disconnected from reality.

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: QUIC traffic throttled on AT&T residential

2020-02-20 Thread Jared Mauch



> On Feb 20, 2020, at 3:30 PM, Daniel Sterling  
> wrote:
> 
> On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch  wrote:
>> if the question is will the browser vendor (google) or the broadband 
>> provider (att)
>> move first, i can already predict the answer.  my experience (again) with 
>> the quic
>> wg is they seem to think there's many options and bad providers will be 
>> replaced
>> which seems disconnected from reality.
> 
> Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where
> there are multiple ISPs.
> 
> But not everyone is, and local-loop unbundling is no longer a thing :(
> 
> The average user is at the mercy of their local ISP. So the major
> providers should (but may not have much incentive to) provide decent
> service.
> 

Honestly, there’s part of me that wants to say be lucky you have a choice.

My options here are (1.5m DSL, fixed wireless ISP behind several layers of NAT,
Cellular data or dial-up).

“Watch this space” for my solution.  Several people know what it is, but
my solution isn’t for everyone.

- Jared



Re: QUIC traffic throttled on AT&T residential

2020-02-20 Thread Jared Mauch



> On Feb 20, 2020, at 4:42 PM, Blake Hudson  wrote:
> 
> 
> 
> On 2/20/2020 1:10 PM, Jared Mauch wrote:
>> On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
>>> On 2/20/2020 10:34 AM, Ca By wrote:
>>>> On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson >>> <mailto:bl...@ispn.net>> wrote:
>>>> Dropping udp is not from a “best practice” doc from a vendor, it is
>>>> deployed by network ops folks that are trying to sleep at night.
>>> I get it Ca, I happen to be one of those network ops folks that likes to
>>> sleep at night. However, I've never thought it was a good practice to break
>>> applications in fun ways for my customers to discover on their own and I've
>>> never sold someone a 150Mbps package that actually only delivers 10Mbps for
>>> certain applications. Regardless of the intent, ATT and Cox's policies are
>>> not transparent, open, or neutral on this topic. This leaves us to speculate
>>> on what their intentions might have been and whether their actions are an
>>> appropriate response to any concerns they might have had.
>>  I was responsible for deploying such policies in the past, going back as
>> far as the UDP/1434 filters I was forced to deploy due to persistent network
>> congestion.  Rolling these back took some time.
>> 
>>  The same is true for UDP policers we ended up rolling out for NTP, 
>> chargen
>> and other activities.
>> 
>>  Extending these to consumer side where the traffic often originated 
>> makes
>> sense until the devices can be secured.  You can blame the providers for 
>> deploying
>> filters, or not disconnecting consumers that have devices that can be 
>> exploited
>> or whatever other reason you believe.
>> 
>>  As a network operator my goal was always to ensure customers receive
>> the traffic they expected, high rates of UDP were often not what they wanted.
>> 
>>  Adusting the limits may be useful but I still think the question of
>> what rate of UDP traffic is acceptable is a practical one for the future.
>> 
>>  - Jared
> 
> I think that's a fair statement Jared. How about this question: Would it be 
> reasonable for one to presume that someone purchasing a 25Mbps internet 
> connection might potentially want to send or receive 25Mbps of UDP traffic? I 
> can think of a few (not uncommon) applications where this would be the case 
> (VPNs, security cameras using RTP, teleconferencing, web browsers 
> implementing QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have 
internet for everyone and internet for the tech-savvy/business split that 
becomes interesting.

I’ve generally been willing to pay more for a business class service for 
support and improved response SLA.  The average user isn’t going to detect that 
10% of their UDP has gone missing, nor should they be expected to.

- Jared

Re: QUIC traffic throttled on AT&T residential

2020-02-20 Thread Jared Mauch



> On Feb 20, 2020, at 4:53 PM, Blake Hudson  wrote:
> 
> 
As a network operator my goal was always to ensure customers receive
 the traffic they expected, high rates of UDP were often not what they 
 wanted.
 
Adusting the limits may be useful but I still think the question of
 what rate of UDP traffic is acceptable is a practical one for the future.
 
- Jared
>>> I think that's a fair statement Jared. How about this question: Would it be 
>>> reasonable for one to presume that someone purchasing a 25Mbps internet 
>>> connection might potentially want to send or receive 25Mbps of UDP traffic? 
>>> I can think of a few (not uncommon) applications where this would be the 
>>> case (VPNs, security cameras using RTP, teleconferencing, web browsers 
>>> implementing QUIC, DNS servers, hosted PBX, etc).
>> I can think of many legitimate cases, but i think this is where you have 
>> internet for everyone and internet for the tech-savvy/business split that 
>> becomes interesting.
>> 
>> I’ve generally been willing to pay more for a business class service for 
>> support and improved response SLA.  The average user isn’t going to detect 
>> that 10% of their UDP has gone missing, nor should they be expected to.
>> 
>> - Jared
> And here I think is where one crosses the threshold between providing an 
> "internet connection" and providing a connection "that can be used to access 
> specific applications or services" (as defined by your provider). This is one 
> step away from your ISP selling you a connection to access Facebook, if you 
> want to access Twitter that's available on their premium package. Oh, you 
> want to access Slack, sorry we don't offer that as a package yet. Call back 
> in a month. You need to esss-esss-achhh? I've never heard of that, why would 
> you want to do that?

AT&T has rarely offered internet service, their required devices for their 
U-Verse often munged traffic.  I recall when you could reboot their boxes by 
sending SIP packets to devices behind them and it would intercept them and 
think it was for itself for their POTS service.

If you have any NAT/ALG in there, it’s not pure internet, but most people want 
access to the “web” and aren’t running ftp/finger/ytalk/uucp/sip etc.. This is 
why SSL VPNs on 443 became a thing over time.

- jared

Re: QUIC traffic throttled on AT&T residential

2020-02-21 Thread Jared Mauch



> On Feb 21, 2020, at 2:22 PM, Dan Wing  wrote:
> 
> There are choices, such as making connection initiation, connection 
> acceptance, and connection termination parsable by network elements on the 
> path so state can be established, maintained, and cleared, DoS can be 
> identified, and so on.  The decision was to hide all that from network 
> elements.

I think those design choices lead directly to these known cases where a UDP 
policer is encountered.  I can drive along the road and not crash into the 
trees, but when I make a choice to deploy something without the proper safety 
equipment and it crashes, blaming the road seems like a poor response.

I can already hear the QUIC WG types blaming the network in abstentia, because 
well, why would an operator want to keep their network functioning? :-)

- Jared

Re: RADB account deletions

2020-03-03 Thread Jared Mauch



> On Mar 3, 2020, at 1:39 PM, Job Snijders  wrote:
> 
> On Tue, Mar 03, 2020 at 11:22:35AM -0700, Clinton Work wrote:
>> It looks like the former Allstream RADB account (MAINT-AS15290) and
>> all associated route objects were removed from RADB today.   The
>> deletion mainly impacts Canadian route objects registered by the
>> former Allstream (now Zayo).   Q9 Networks MAINT-AS12188 was deleted
>> at the same time.   
> 
> For those interested, I think this is roughly what was deleted:
> 
> snapshot/20200302-0001/whois.radb.net$ awk 'BEGIN {RS=""; FS="\n"} 
> /MAINT-AS15290/ {print $0 "\n"}' radb.db > MAINT-AS15290.2020.03.02.txt
> 
> file @ http://instituut.net/~job/MAINT-AS15290.2020.03.02.txt
> 
> I'm sure RADB staff can restore was appropriate if they are contacted.
> 

I know that ARIN had some IRR related issues in the last week, but you may want 
to move objects there if feasible.

- Jared



Re: RADB account deletions

2020-03-04 Thread Jared Mauch



> On Mar 4, 2020, at 12:27 PM, Clinton Work  wrote:
> 
> RADB MAINT-AS15290 was restored this morning along with all the original 
> route objects.   The proxy route objects I added Tuesday were replaced by the 
> original MAINT-AS15290 objects.   


I’m curious if there’s a reason that objects are not being placed in the ARIN 
IRR?  It’s free to use to ARIN members and is mirrored by all the major players.

- Jared

Re: Hi-Rise Building Fiber Suggestions

2020-03-09 Thread Jared Mauch



> On Feb 26, 2020, at 12:42 PM, Randy Bush  wrote:
> 
> since we're at this layer, should i worry about going 3m with dacs at
> low speed, i.e. 10g?  may need to do runs to neighbor rack.


No.

We even do this for 100G.

- Jared


Work from Home and other dynamics

2020-03-09 Thread Jared Mauch
I’m wondering what general trends people have seen with the recent reduction in 
travel and increased work from home activities.

I’m expecting that a number of networks are seeing increased traffic demand.  
Enterprises are likely adding VPN licenses for staff that are now remote, etc..

I would expect increase (and decreases) similar to weekend traffic patterns. 

I’m expecting there will be more IPv6 traffic similar to what is seen during 
the Christmas/New Years holiday on this traffic as well:

https://www.google.com/intl/en/ipv6/statistics.html

What interesting dynamics are you seeing?

- Jared




Re: akamai yesterday - what in the world was that

2020-03-10 Thread Jared Mauch



> On Mar 10, 2020, at 6:14 PM, Bryan Holloway  wrote:
> 
> We hit over 40G on one of our PNIs.
> 
> Currently, however, I'm trying to figure out why we're still seeing a 
> significant amount of traffic over transit when we have PNIs at the same 
> locations ...
> 
> I've reached out to Akamai, but I haven't heard anything back yet. I'm sure 
> they're busy …

Yes another high traffic day.  I’ll get with you in private about your specific 
situation.

As always, there’s bumps but if we can improve things I want to do that.


- Jared



Re: COVID-19 vs. our Networks

2020-03-12 Thread Jared Mauch



> On Mar 12, 2020, at 3:00 PM, Troy Martin  wrote:
> 
> On March 12, 2020 10:22 AM, g...@1337.io wrote: 
>> With talk of there being an involuntary statewide (WA) and then national
>> quarantines (house arrest) for multiple weeks, has anyone put thought
>> into the impacts of this on your networks if/when this comes to
>> fruition?
> 
> No WFH policy here yet, but I've unpacked a couple of our ASAs from before our
> most recent edge refresh to run AnyConnect for some of our remote sites. I'm
> not expecting too much load being that we're not a particularly large company,
> but it's better to have an extra couple RU filled up and happily whirring away
> than to get a phone call at 5am when everyone in Eastern Time logs in and the
> throughput drops like the Dow Jones.
> 
> I'm more worried about the lead times on new hardware skyrocketing than the
> impact of having 8-10x the teleworkers. At least we can still fulfill orders
> for software licenses…

We are all on a split VPN here and slowly moving to a no-VPN solution with some 
users already on that.  They are finding things that aren’t quite covered right 
or properly, but that list is slowly shrinking.

I’m expecting that many places will be moving from a full VPN to a split 
solution.  At my prior employer we did split VPN as well for v4 and full for V6 
as not everyone had native v6, so split didn’t make sense there.  (Those with 
native v6 did gripe, but it was better to have a consistent solution).

I’m expecting that despite the usual game and download/streaming events, the 
baseline usage during the daytime is going to tick up significantly eating into 
network margin.  Hopefully everyone has your upgrades on order due to the 
aforementioned lead-time issues.  Hopefully everything is back to normal in 4-6 
weeks.

I’m looking forward to a few people learning how to WFH and expecting many 
people to realize how much they don’t get along as well when they’re in the 
house all day with the kids.  

We have an internal thread going on the WFH tips:

#1 Take a break.  That walk you would take to Starbucks or whatever, build 
something comparable into it at home.

There’s a few other pro-tips, but the take a break one is one I feel is 
important.  

- Jared



Re: AT&T is suspending broadband data caps for home internet customers due to coronavirus

2020-03-12 Thread Jared Mauch
I do worry if the broadband networks have the capacity. WFH traffic is usually 
different from regular consumer traffic. My neighbors were telling me about the 
mandatory work from home they had today and how the VPN struggled to work. 

To those upgrading those things, keep at it. You will get there. 

Sent from my iCar

> On Mar 12, 2020, at 6:29 PM, Sean Donelan  wrote:
> 
> 
> The first data cap waiver I've seen due to coronavirus.  I expect other ISPs 
> to quickly follow.
> 
> https://www.vice.com/en_us/article/v74qzb/atandt-suspends-broadband-usage-caps-during-coronavirus-crisis
> 
> AT&T is the first major ISP to confirm that it will be suspending all 
> broadband usage caps as millions of Americans bunker down in a bid to slow 
> the rate of COVID-19 expansion. Consumer groups and a coalition of Senators 
> are now pressuring other ISPs to follow suit.


Re: AT&T is suspending broadband data caps for home internet customers due to coronavirus

2020-03-12 Thread Jared Mauch



> On Mar 12, 2020, at 6:55 PM, Sean Donelan  wrote:
> 
> On Thu, 12 Mar 2020, Tom Paseka wrote:
>> I am not worried. Residential ISPs are usually at peak in the late evening.
>> They have loads of capacity during the day.
> 
> Do they have capacity to the right places?
> 
> The evening traffic peak is usually streaming entertainment bits from CDNs on 
> the edge of the network.
> 
> Work From Home seems to be VPNs going between residential to business ISPs 
> interconnections and lots of videoconferencing services (zoom, webex, etc).

Yes, this is what I’m concerned about.  Most of the content/cloud people have 
built networks around the capacity needed to get bits into the networks and 
often aggressively peer.

The corporate office that is behind one incumbent that now has a global set of 
people doing VPN activities at 10x the prior capacity of a week ago may have a 
harder time fitting.

I expect there will be areas which see a higher base load that contributes to 
seeing the peaks earlier.

- Jared

Re: DHS letters for fuel and facility access

2020-03-16 Thread Jared Mauch



> On Mar 16, 2020, at 4:24 PM, james jones  wrote:
> 
> Fuel priority? They expecting shortage and/or power outages?
> 

I suspect it’s more to solve issues with truck drivers going to work and their 
job is to deliver fuel.  Some areas have been instituting curfews and this 
would satisfy the local authorities who may stop such a driver.

- Jared



UDP/123 policers & status

2020-03-17 Thread Jared Mauch
I’m curious what people are seeing these days on the UDP/123 policers in their 
networks.

I know while I was at NTT we rolled some out, and there are a number of 
variants that have occurred over the past 6-7 years.  I’ve heard from people at 
the NTP Pool as well as having observed some issues with NTP at Akamai and time 
sync from time to time.

Are you still seeing a lot of NTP attacks in your flows these days?

Should we be looking to remove these, similar to how we did for SQL/Slammer 
after a time?

- Jared

Re: interesting troubleshooting

2020-03-20 Thread Jared Mauch



> On Mar 20, 2020, at 5:50 PM, Job Snijders  wrote:
> 
> On Fri, Mar 20, 2020 at 05:33:31PM -0400, Nimrod Levy wrote:
>> With the increase in remote workers and VPN traffic that won't hash across
>> multiple paths, I thought this anecdote might help someone else track down
>> a problem that might not be so obvious.
> 
> Do we know which specific VPN technologies specifically are harder to
> hash in a meaningful way for load balanacing purposes, than others?
> 
> If the outcome of this troubleshooting is a list of recommendations
> about which VPN approaches to use, and which ones to avoid (because of
> the issue you described), that'll be a great outcome.
> 

It’s the protocol 50 IPSEC VPNs.  They are very sensitive to path changes and 
reordering as well.

If you’re tunneling more than 5 or 10Gb/s of IPSEC it’s likely going to be a 
bad day when you find a low speed link in the middle.  Generally providers with 
these types of flows have both sides on the same network vs going off-net as 
they’re not stable on peering links that might change paths.

You also need to watch out to ensure you’re not on some L2VPN type product that 
bumps up against a barrier.  I know it’s a stressful time for many networks and 
systems people as traffic shifts.  Good luck out there!

- Jared



Re: NTT/AS2914 enabled RPKI OV 'invalid = reject' EBGP policies

2020-03-26 Thread Jared Mauch



> On Mar 25, 2020, at 10:05 PM, JASON BOTHE via NANOG  wrote:
> 
> Excellent work. I’m curious to know how many of the big ASs are participating 
> to date. If you or anyone on the list knows if this is published please let 
> me know. 

Quite a number have done this.  I expect we are getting close to herd immunity 
if a few more large providers are able to deploy.

- Jared

Re: Backhoe season?

2020-03-31 Thread Jared Mauch



> On Mar 27, 2020, at 2:32 AM, Hank Nussbacher  wrote:
> 
> On 26/03/2020 20:02, Aaron Gould wrote:
> 
> Numerous gov'ts and municipalities, which had planned constructions jobs but 
> postponed them to the summer due to heavy traffic volume, have started to 
> implement all those construction jobs, which includes backhoes.
> 

Around here this is what is deemed essential:

- snip - 

All gas, electric, telcom, fiber, and MDOT projects are considered "critical 
infrastructure" and essential under Governor Whitmer's Executive Order 2020-21. 
ONLY those doing work essential to the needs of our state and our 
infrastructure should be placing tickets.

- snip -

This means that my fiber construction is ongoing.  Plus those crews shouldn’t 
get closer than 6’ from each other in most cases.

- Jared

Re: FCC grants WISPs temporary 5.9 GHz spectrum access

2020-04-01 Thread Jared Mauch
The big announcement is the 6ghz space opening up. This will be big for people 
doing p2p links. 

Sent from my iCar

> On Apr 1, 2020, at 8:42 PM, Sean Donelan  wrote:
> 
> 
> I missed this announcement last week.
> 
> 
> https://www.fcc.gov/document/fcc-grants-wisps-temporary-59-ghz-spectrum-access-rural-broadband
> 
> The FCC’s Wireless Telecommunications Bureau today granted temporary spectrum 
> access to 33 wireless Internet service providers serving 330
> counties in 29 states to help them serve rural communities facing an increase 
> in broadband needs during the COVID-19 pandemic. The Special Temporary 
> Authority (STA) granted today allows these companies to use the lower 45 
> megahertz of spectrum in the 5.9 GHz band for 60 days.


Re: Spike in traffic to Google&Akamai caches?

2020-04-21 Thread Jared Mauch



> On Apr 21, 2020, at 8:56 AM, Hank Nussbacher  wrote:
> 
> Did anyone notice a huge jump in traffic today between 11:30-11:40 (GMT) 
> directed at Google and Akamai caches coming from Amazon and Google?
> Gaming updates?
> 

I’m not seeing any big spikes in our graphs, but what you are describing may be 
the caches going forward to our customer origin or similar.

If anyone else is seeing something similar or has concerns you can reach me 
off-list.

Hank, can you send me the IPs of your cluster in private to speed up triage?

- Jared



Re: Are underground utility markers essential workers?

2020-04-21 Thread Jared Mauch
USIC is not marking on Fridays around here.

URG is marking but only 4 hours a day.

- Jared

> On Apr 21, 2020, at 4:44 PM, Jeff Shultz  wrote:
> 
> Since in our case they are Outside Plant Tech's who are assigned the
> duties as needed, they are essential workers.
> 
> On Tue, Apr 21, 2020 at 11:59 AM Sean Donelan  wrote:
>> 
>> 
>> Utility markers don't get the recognition they deserve.  If they aren't
>> essential workers, they should be and get hazard pay.
>> 
>> They help protect everyone's fiber and cables and pipes that go boom.
>> 
>> 
> 
> 
> -- 
> Jeff Shultz
> 
> -- 
> Like us on Social Media for News, Promotions, and other information!!
> 
>
>   
>   
>   
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _ This message 
> contains confidential information and is intended only for the individual 
> named. If you are not the named addressee you should not disseminate, 
> distribute or copy this e-mail. Please notify the sender immediately by 
> e-mail if you have received this e-mail by mistake and delete this e-mail 
> from your system. E-mail transmission cannot be guaranteed to be secure or 
> error-free as information could be intercepted, corrupted, lost, destroyed, 
> arrive late or incomplete, or contain viruses. The sender therefore does 
> not accept liability for any errors or omissions in the contents of this 
> message, which arise as a result of e-mail transmission. _



Re: Google peering pains in Dallas

2020-04-30 Thread Jared Mauch



> On Apr 29, 2020, at 7:59 PM, Kaiser, Erich  wrote:
> 
> So it has been 3 weeks of major ICMP packet loss to any google service over 
> the Dallas Equinix IX, it is not affecting performance of service but is 
> affecting us with customer complaints and service calls due to some software 
> using it for monitoring purposes people using it for benchmark testing.  I 
> have been told from them that they know the cause now and know that a Large 
> ISP on the IX is causing the issue(Hmm wonder who that is...), so why do they 
> not shutdown the peer with them and force the ISP to fix the issue?  This 
> issue is affecting everyone on the IX not just us, very very frustrating.  
> Hopefully this will reach someone over there that can do something about it….

Issues with the IXP ecosystem aren’t new in the US.  This is why some providers 
don’t appear at them.  The original one member could hurt it all was really the 
gigaswitch HOLB (head of line blocking) issue that was triggered by congested 
ports.

(Waits for others to crawl out of the woodwork who were more involved in this 
:-) 

This is why the majority of traffic volume for interconnection has generally 
been over private peering links (paid, SFI, otherwise).

If you tried to force it through an IXP ecosystem the tens of Tbps wouldn’t fit 
even in each city.  Things like CDNs, the Netflix OpenConnect and otherwise 
have really shifted the demand off the interconnection points as much as 
feasible.  Sometimes an organization can’t handle it or tries to cling to it’s 
old ways.  Sometimes it takes organization change or people change to improve 
the situation.

I know it can sound like a broken record, but upgrading to match the capacity 
demands really can make a difference to offload paths.  It may also expose 
other weak points.  My personal goal is to cease thinking about things in the 
95/5 model and more of a peak model.  95/5 gets you so far but the peaks are 
really where networks can shine or show their age.

I understand it’s not always possible to upgrade links, or sometimes one party 
holds out on the other.  It’s certainly not the case at $dayjob and I try to 
ensure the process works as best as it can here.  

Sometimes it’s best to just de-peer a network.  You may find it works out 
better for all involved.

At $nightJob I want to peer as much traffic off as possible, but if the network 
paths aren’t there or low-speed it may not make sense.

Evaluate your peers periodically to ensure you are getting what you expect.

- jared

Re: Google peering pains in Dallas

2020-04-30 Thread Jared Mauch



> On Apr 30, 2020, at 2:41 PM, Christopher Morrow  
> wrote:
> 
> It sounds like, to me anyway, you'd like to copy/paste/sed the AS112
> project's goals, no?

I just use this page:

http://hasthelargehadroncolliderdestroyedtheworldyet.com/

- jared

Re: Did I miss a problem: FCC and CISA stress need for access during pandemic

2020-05-27 Thread Jared Mauch
I have had problems with OSP construction ostensibly delayed by closed 
permitting agencies. 

Sent from my iCar

> On May 27, 2020, at 2:30 AM, Bill Woodcock  wrote:
> 
> 
> 
>> On May 27, 2020, at 3:40 AM, Sean Donelan  wrote:
>> I have not heard of any problems with access for ISP and communications 
>> workers in any U.S. state or locality during the pandemic.
>> Did I miss a big problem requiring the FCC chairman and CISA Director send a 
>> letter?
> 
> That was one of the outcomes of the OECD recommendations to member 
> governments on the Internet during the pandemic.  As you may recall, I 
> emailed you, and many other members of our community, on March 23, soliciting 
> input for this document:
> 
>   
> http://www.oecd.org/coronavirus/policy-responses/keeping-the-internet-up-and-running-in-times-of-crisis-4017c4c9/
> 
> The specific recommendation regarding prioritized access came from several of 
> the people I mailed, and was of particular concern to global backbone 
> operators. Whether you think that particular recommendation is a high 
> priority or not, I’d chalk this up as a successful exercise of our community 
> providing input to government and having government take it seriously and act 
> upon it in the way that we requested them to.  Exercising that channel 
> periodically, to keep government thinking of that as normal, would be good.
> 
> There’s no provable causality chain here, but it was a concern, we spoke, 
> they listened, and the problem we were concerned with did not become an 
> issue, so that’s a success.  If only we could do that with public health, 
> we’d be in great shape.
> 
>   -Bill
> 
> 
> 
> 
>-Bill
> 


Re: non-rate limited, automatable Looking Glasses?

2020-07-18 Thread Jared Mauch
And you can also use ripe atlas as well. If you need credits ask on that list 
and people offer them up regularly and quickly. 

Sent from my iCar

> On Jul 18, 2020, at 5:45 PM, Brendan Halley  wrote:
> 
> 
> Hi Lars,
> 
> You should check out https://ring.nlnog.net/ by contributing resources 
> yourself you also get access to a wide array of machines from all across the 
> world you can use to turn traceroutes and pings.
> 
> Some wrappers have already been made to run commands against multiple 
> machines at the same time (https://ring.nlnog.net/toolbox/), you'll have SSH 
> access to run any commands you want and there is an API to find the probes if 
> you want to automate it all.
> 
> I encourage anyone and everyone to join. The more networks the better!
> 
> Brendan 
> 
>> On Sun, 19 Jul 2020, 7:36 am Lars Prehn,  wrote:
>> Hi everyone,
>> 
>> In the next couple of months, I want to compare data plane and control 
>> plane measurements on a larger scale. In particular, I'm looking for 
>> (publicly accessible) devices that receive BGP feeds and can perform a 
>> bunch of automated (paris) traceroutes. I currently do not have an exact 
>> probing rate or target set in mind; however, I'm sure that manually 
>> entering IP addresses as targets for usual Looking glasses won't cut it. 
>> Does anyone know less-restricted (maybe even automatable?) Looking 
>> Glasses (or similar devices) or is willing to provide access to one?
>> 
>> BTW: I though about picking Atlas probes from ASes that feed BGP 
>> Collector Projects (e.g. RIPE RIS or RouteViews). Unfortunately, the 
>> respective probes are often really far apart from the feeding routers; 
>> thus, their individual perspectives are likely misaligned :(
>> 
>> Best regards,
>> 
>> Lars
>> 


Re: Project/Tool to deploy and maintain Edge Servers (VMs) remotely

2020-09-04 Thread Jared Mauch



> On Sep 4, 2020, at 6:52 AM, Douglas Fischer  wrote:
> 
> I'm looking for some tool to work as a Comand and Control of several remote 
> nodes.
> 
> The idea is to have many-many nodes of Virtual Machines running on every ASN 
> voluntarily to deploy some services spread everywhere we can.
> 
> 
> Something like a Call-Home, that allows the headquarter to track operation, 
> health state, deploy commands.
> (Re-reading this phrase, it could sound like a newbie
>  hacker trying to create his own Mirai. haha...
>  It's not the case... I'm not on the dark side of the force.)
> 
> 
> I was thinking in use something like reverse ssh and ansible.
> But I thought that I'm probably reinventing part of the wheel.
> 
> I want to believe that there already some projects that would put-me some 
> steps further on this project.
> 
> 
> Could anyone give-me some-tip?
> 

You may want to look at the NLNOG Ring both for examples, but also for how to 
have interesting views from around the world, similar to RIPE Atlas.

- Jared



Re: Getting Fiber to My Town by Jared Mauch

2020-09-10 Thread Jared Mauch



> On Sep 10, 2020, at 8:06 AM, Jared Brown  wrote:
> 
> I believe this belongs here:
> 
> Getting Fiber to My Town by Jared Mauch
> https://www.youtube.com/watch?v=ASXJgvy3mEg (YouTube video of NLnog 
> presentation)
> https://nlnog.net/static/live/nlnog_live_sep_2020_jared.pdf (slides for 
> presentation)
> https://news.ycombinator.com/item?id=24424910#24430901 (discussion on Hacker 
> News with Jared participating)
> https://washftth.com/ (project homepage)
> 
> I find this an interesting description of how to apply skills that we 
> normally only use at work to solve connectivity issues at home. Quite timely 
> too, as home connectivity is needed more than ever.

As my kids are in the other room on 4x zooms at once, the prior connection 
could not have survived the load with them and me working as well.

I’m working with the PC on giving another version of this talk (I want to 
provide more financial details) for an upcoming meeting.

Stay tuned to the NANOG Agenda :-)

- Jared

Re: Getting Fiber to My Town by Jared Mauch

2020-09-10 Thread Jared Mauch



> On Sep 10, 2020, at 4:10 PM, Jared Geiger  wrote:
> 
> Another Jared with a question. What method did you use to blow the fiber 
> through the conduit? You mentioned you had trouble figuring out the process 
> relating to lubrication and building a contraption to blow the fiber.

You need the conduit lube.  The duraline summer blowing lube worked well for me.

- Jared

Re: Getting Fiber to My Town by Jared Mauch

2020-09-10 Thread Jared Mauch
The pulling lube does not work well IMO, the blowing lube made a huge 
difference.

- Jared

> On Sep 10, 2020, at 4:29 PM, Josh Luthman  wrote:
> 
> I believe this is the stuff we used on our project:
> https://www.menards.com/main/electrical/electrical-tools-accessories/wire-conduit-installation/ideal-regyellow-77-wire-pulling-lubricant-5-gallon/31-355/p-133962344-c-6458.htm
>   
> 
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> 
> On Thu, Sep 10, 2020 at 4:25 PM Jared Mauch  wrote:
> 
> 
> > On Sep 10, 2020, at 4:10 PM, Jared Geiger  wrote:
> > 
> > Another Jared with a question. What method did you use to blow the fiber 
> > through the conduit? You mentioned you had trouble figuring out the process 
> > relating to lubrication and building a contraption to blow the fiber.
> 
> You need the conduit lube.  The duraline summer blowing lube worked well for 
> me.
> 
> - Jared



Re: BFD for routes learned trough Route-servers in IXPs

2020-09-22 Thread Jared Mauch



> On Sep 22, 2020, at 4:46 AM, Andy Davidson  wrote:
> 
> Hi,
> 
> Douglas Fisher wrote:
>> B) There is any other alternative to that?
> 
> Don't connect to IXPs with very very large and complicated topologies. 
> Connect to local IXPs where the design makes a forwarding plane failure that 
> causes the problem you describe less likely. 

Or don’t use a route server except to bootstrap.  I regularly see issues 
related to them.  I get it’s not easy to peer at an IXP, but IXP peering isn’t 
for everyone as some people might make it sound.  This is why back in the day 
there was a push to require 24x7 staffing of the remote side to ensure it was 
being monitored/supported.

That may no longer apply to many people, but without active monitoring, you 
won’t know what the state is of the remote side.

- Jared

Re: Gaming Consoles and IPv4

2020-09-28 Thread Jared Mauch
I’m still waiting for my ISP to turn on v6 so the consumers of my neighborhood 
ISP can get v6 service.

Going to poke them again today actually.

- Jared

> On Sep 28, 2020, at 8:37 AM, Justin Wilson (Lists)  wrote:
> 
> It is coming back to that, but you still have so much going on that you need 
> the open ports.  I don’t gt why people fight IPV6 so much.  
> 
> 
> Justin Wilson
> j...@mtin.net
> 
> —
> https://j2sw.com - All things jsw (AS209109)
> https://blog.j2sw.com - Podcast and Blog
> 
>> On Sep 28, 2020, at 8:34 AM, Mike Hammett  wrote:
>> 
>> Why stray away from how PC games were 20 years ago where there was a 
>> dedicated server and clients just spoke to servers?
>> 
>> 
>> 
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> 
>> Midwest Internet Exchange
>> 
>> The Brothers WISP
>> 
>> From: "Justin Wilson (Lists)" 
>> To: "North American Network Operators' Group" 
>> Sent: Monday, September 28, 2020 7:22:28 AM
>> Subject: Re: Gaming Consoles and IPv4
>> 
>> There are many things going on with gaming that makes natted IPv4 an issue 
>> when it comes to consoles and gaming in general.   When you break it down it 
>> makes sense.
>> 
>> -You have voice chat
>> -You are receiving data from servers about other people in the game
>> -You are sending data to servers about yourself
>> -If you are using certain features where you are “the host” then you are 
>> serving content from your gaming console.  This is not much different than a 
>> customer running a web server.  You can’t have more than one customer 
>> running a port 80 web-server behind nat.
>> -Streaming to services like Twitch or YouTube
>> 
>> All of these take up standard, agreed upon ports. It’s really only prevalent 
>> on gaming consoles because they are doing many functions.  Look at it 
>> another way.  You have a customer doing the following.
>> 
>> -Making a VOIP call
>> -Streaming a movie
>> -Running a web server
>> -Running bittorrent on a single port
>> -Having a camera folks need to access from the outside world
>> 
>> This is why platforms like Xbox developed things like Teredo.
>> 
>> Justin Wilson
>> j...@mtin.net
>> 
>> —
>> https://j2sw.com - All things jsw (AS209109)
>> https://blog.j2sw.com - Podcast and Blog
>> 
>> On Sep 27, 2020, at 9:33 PM, Daniel Sterling  
>> wrote:
>> 
>> Matt Hoppes raises an interesting question,
>> 
>> At the risk of this being off-topic, in the latest call of duty games I've 
>> played, their UDP-NAT-breaking algorithm seems to work rather well and 
>> should function fine even behind CGNAT. Ironically turning on upnp makes 
>> this *worse*, because when their algorithm probes to see what ports to use, 
>> upnp sends all traffic from the "magical xbox port" to one box instead of 
>> letting NAT control the ports. This does cause problems when multiple xboxes 
>> are behind one NAT doing upnp. If upnp is on and both xboxes are fully 
>> powered off and then turned on one at a time, things do work. But when upnp 
>> is off everything works w/o having to do that.
>> 
>> There are many other games and many CPE NAT boxes that may do horrible 
>> things, but CGNAT by itself shouldn't cause problems for any recent device / 
>> gaming system.
>> 
>> It is true that I've yet to see any FPS game use ipv6. I assume that's cuz 
>> they can't count on users having v6, so they have to support v4, and it 
>> wouldn't be worth their while to have their gaming host support dual-stack. 
>> just a guess there
>> 
>> -- Dan
>> 
>> 
>> 
>> On Sun, Sep 27, 2020 at 7:29 PM Mike Hammett  wrote:
>> Actually, uPNP is the only way to get two devices to work behind one public 
>> IP, at least with XBox 360s. I haven't kept up in that realm.
>> 
>> 
>> 
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> 
>> Midwest Internet Exchange
>> 
>> The Brothers WISP
>> 
>> From: "Matt Hoppes" 
>> To: "Darin Steffl" 
>> Cc: "North American Network Operators' Group" 
>> Sent: Sunday, September 27, 2020 1:22:51 PM
>> Subject: Re: Gaming Consoles and IPv4
>> 
>> I understand that. But there’s a host of reasons why that night not work - 
>> two devices trying to use UPNP behind the same PAT device, an apartment 
>> complex or hotel WiFi system, etc. 
>> 
>> On Sep 27, 2020, at 2:17 PM, Darin Steffl  wrote:
>> 
>> 
>> This isn't rocket science.
>> 
>> Give each customer their own ipv4 IP address and turn on upnp, then they 
>> will have open NAT to play their game and host. 
>> 
>> On Sun, Sep 27, 2020, 12:50 PM Matt Hoppes 
>>  wrote:
>> I know the solution is always “IPv6”, but I’m curious if anyone here knows 
>> why gaming consoles are so stupid when it comes to IPv4?  
>> 
>> We have VoIP and video systems that work fine through multiple layers of PAT 
>> and NAT. Why do we still have gaming consoles, in 2020, that can’t find 
>> their way through a PAT system with STUN or other methods?
>> 
>> It seems like this should be a simple solution, why are we still opening 
>> ports or having systems that don’

Re: telia selling carrier ops to polhem infra

2020-10-06 Thread Jared Mauch



> On Oct 6, 2020, at 11:28 AM, James Breeden  wrote:
> 
> Still smells Swedish to me. Probably will end up with a different name, but 
> other than that I don't see much changing. Sounds more like a spinoff than 
> acquisition.


Yeah, I guess my questions are more long-term, usually in the 12 month window 
not much changes, but I’ve seen people expand to be global carriers then shrink 
back to their incumbent countries, (eg: 5511, 3320) over the years with a few 
holding their own, or having some minimal international presence..

What will happen here?  I’m curious as there are usually some economic drivers 
of these events.  I’m not sure I see what would be a driver unless the business 
is losing significant amount of SEK and this is effectively a restructuring.

- Jared

9500 BGP Contact

2021-09-07 Thread Jared Mauch
Hello,

I’m looking for someone at 9500 that can look into why you started announcing 
some IP space starting on August 25th intermittently that is causing an 
operational network issue.

Can you please contact me off-list?

Thanks

- jared

Re: do bgp optimizers think?

2021-09-09 Thread Jared Mauch



> On Sep 9, 2021, at 11:44 AM, Randy Bush  wrote:
> 
> to control inbound traffic, how do bgp optimizers decide how to tune
> what they announce?  slfow?  exploration?  ouija board?
> 
> randy


Generally via sFlow or other traffic detail models.

- Jared


Re: IPv6 woes - RFC

2021-09-16 Thread Jared Mauch



> On Sep 16, 2021, at 9:23 AM, Ca By  wrote:
> 
> 
> 
> 
> This has nothing to do with IPv6, of course, other than that modern phones use
> VoLTE so within a mobile carrier's network your voice call is probably handled
> using IPv6 transport.
> 
> Good point John. 
> 
> A lot of folks missed that ipv6 absorbed the scale growth in mobile, and 
> mobile is what most eyeballs and be big content consider the internet to be. 
> And, yes, mobile voice is called VoLTE and is most commonly deployed in the 
> usa with ipv6. 
> 
> And most internet is called youtube / fb, and that is ipv6 too. 
> 
> This is where i live and work , 87% of mobiles on v6, voice and data
> 
> https://www.worldipv6launch.org/apps/ipv6week/measurement/images/graphs/CombinedUSMobileCarriers.png
> 
> This is where nanog seems to be (old man yells at cloud meme)
> 
> https://static.wikia.nocookie.net/memepediadankmemes/images/0/01/297.jpg/revision/latest?cb=20180908193511
> 
> I don’t see the failure of ipv6 in 2021. It is globally deployed, providing 
> global address, to billions of things and PB/s of content 
> 
> There are laggards in adopting v6, but they have not stopped the frontier of 
> internet to reaching billions of people and things. 
> 


Yeah, I think this is the thing that I see people most often missing.  Yes your 
provider may not be doing IPv6, but many applications and providers may yet be 
IPv6 internally or IPv6 to the popular content. 

I also say the number of people who store an IP as integer in a mysql(mariadb) 
backend is not to be underestimated.  

I still see some people doing the split up the IPv6 to store it in multiple 
columns thing even in 2021 which is disappointing as it shows this 
backwards/legacy thinking.

If you’re seeing majority IPv4 bits, you likely have some choke point (CGN/FW) 
or similar gateway on the content side that needs a major update.  I saw a post 
yesterday that someone used a unicode char to add an account nickname at their 
bank and it caused the entire bank to pause and them to get a phone call.  Not 
sure if it’s true, but it rings true.

Be the one enabling the next-generation access and you’ll similarly see graphs 
like the mobile carriers see.

- Jared

Re: IPv6 woes - RFC

2021-09-18 Thread Jared Mauch
I mostly agree with this. Even new hardware like eero doesn't do v6 by default. 
It's just off. So many things are like this. It's nice that LTE and other 
deployments have v6 by default. Last time I knew providers like t mobile are 
great but their MVNOs like Ultra Mobile do not do v6. 

All this and we will get there. I'm expecting another decade or two though. 

Sent from my TI-99/4a

> On Sep 18, 2021, at 4:29 PM, John R. Levine  wrote:
> 
> IPv4 isn't going away any time soon.


Disaster Recovery Process

2021-10-05 Thread Jared Mauch



> On Oct 4, 2021, at 4:53 PM, Jorge Amodio  wrote:
> 
> How come such a large operation does not have an out of bound access in case 
> of emergencies ???
> 
> 

I mentioned to someone yesterday that most OOB systems _are_ the internet.  It 
doesn’t always seem like you need things like modems or dial-backup, or access 
to these services, except when you do it’s critical/essential.

A few reminders for people:

1) Program your co-workers into your cell phone
2) Print out an emergency contact sheet
3) Have a backup conference bridge/system that you test
  - if zoom/webex/ms are down, where do you go?  Slack?  Google meet? Audio 
bridge?
  - No judgement, but do test the system!
4) Know how to access the office and who is closest.  
  - What happens if they are in the hospital, sick or on vacation?
5) Complacency is dangerous
  - When the tools “just work” you never imagine the tools won’t work.  I’m 
sure the lessons learned will be long internally.  
  - I hope they share them externally so others can learn.
6) No really, test the backup process.



* interlude *

Back at my time at 2914 - one reason we all had T1’s at home was largely so we 
could get in to the network should something bad happen.  My home IP space was 
in the router ACLs.  Much changed since those early days as this network became 
more reliable.  We’ve seen large outages in the past 2 years of platforms, 
carriers, etc.. (the Aug 30th 2020 issue is still firmly in my memory).  

Plan for the outages and make sure you understand your playbook.  It may be 
from snow day to all hands on deck.  Test it at least once, and ideally with 
someone who will challenge a few assumptions (eg: that the cell network will be 
up)

- Jared

Re: Disaster Recovery Process

2021-10-05 Thread Jared Mauch



> On Oct 5, 2021, at 10:05 AM, Karl Auer  wrote:
> 
> On Tue, 2021-10-05 at 08:50 -0400, Jared Mauch wrote:
>> A few reminders for people:
>> [excellent list snipped]
> 
> I'd add one "soft" list item:
> 
> - in your emergency plan, have one or two people nominated who are VERY
> high up in the organisation. Their lines need to be open to the
> decisionmakers in the emergency team(s). Their job is to put the fear
> of a vengeful god into any idiot who tries to interfere with the
> recovery process by e.g. demanding status reports at ten-minute
> intervals.

At $dayjob we split the technical updates on a different bridge from the 
business updates.

There is a dedicated team to coordinate an entire thing, they can be low 
severity (risk) or high severity (whole business impacting).

They provide the timeline to next update and communicate what tasks are being 
done.  There’s even training on how to be a SME in the environment.  

Nothing is perfect but this runs very smooth at $dayjob

- Jared



Re: DNS pulling BGP routes?

2021-10-06 Thread Jared Mauch
This is quite common to tie an underlying service announcement to BGP 
announcements in an Anycast or similar environment.  It doesn’t have to be 
externally visible like this event for that to be the case.

I would say more like Application availability caused the BGP routes to be 
withdrawn.  

I know several network operators that run DNS internally (even on raspberry pi 
devices)  and may have OSPF or BGP announcements internally to ensure things 
work well.  If the process dies (crash, etc) they want to route to the next 
nearest cluster.

Of course if they all are down there’s negative outcomes.

- Jared


> On Oct 6, 2021, at 1:42 PM, Michael Thomas  wrote:
> 
> So if I understand their post correctly, their DNS servers have the ability 
> to withdraw routes if they determine are sub-optimal (fsvo). I can certainly 
> understand for the DNS servers to not give answers they think are unreachable 
> but there is always the problem that they may be partitioned and not the 
> routes themselves. At a minimum, I would think they'd need some consensus 
> protocol that says that it's broken across multiple servers.
> 
> But I just don't understand why this is a good idea at all. Network topology 
> is not DNS's bailiwick so using it as a trigger to withdraw routes seems 
> really strange and fraught with unintended consequences. Why is it a good 
> idea to withdraw the route if it doesn't seem reachable from the DNS server? 
> Give answers that are reachable, sure, but to actually make a topology 
> decision? Yikes. And what happens to the cached answers that still point to 
> the supposedly dead route? They're going to fail until the TTL expires anyway 
> so why is it preferable withdraw the route too?
> 
> My guess is that their post while more clear that most doesn't go into enough 
> detail, but is it me or does it seem like this is a really weird thing to do?
> 
> Mike
> 
> 
> On 10/5/21 11:56 PM, Bjørn Mork wrote:
>> Masataka Ohta  writes:
>> 
>>> As long as name servers with expired zone data won't serve
>>> request from outside of facebook, whether BGP routes to the
>>> name servers are announced or not is unimportant.
>> I am not convinced this is true.  You'd normally serve some semi-static
>> content, especially wrt stuff you need yourself to manage your network.
>> Removing all DNS servers at the same time is never a good idea, even in
>> the situation where you believe they are all failing.
>> 
>> The problem is of course that you can't let the servers take the
>> decision to withdraw from anycast if you want to prevent this
>> catastrophe.  The servers have no knowledge of the rest of the network.
>> They only know that they've lost contact with it.  So they all make the
>> same stupid decision.
>> 
>> But if the servers can't withdraw, then they will serve stale content if
>> the data center loses backbone access. And with a large enough network
>> then that is probably something which happens on a regular basis.
>> 
>> This is a very hard problem to solve.
>> 
>> Thanks a lot to facebook for making the detailed explanation available
>> to the public.  I'm crossing my fingers hoping they follow up with
>> details about the solutions they come up with.  The problem affects any
>> critical anycast DNS service. And it doesn't have to be as big as
>> facebook to be locally critical to an enterprise, ISP or whatever.
>> 
>> 
>> 
>> Bjørn



Re: IPv6 and CDN's

2021-10-25 Thread Jared Mauch
On Fri, Oct 22, 2021 at 05:13:09PM +0200, Job Snijders via NANOG wrote:
> Hi everyone, goedenmiddag Marco!
> 
> On Fri, Oct 22, 2021 at 01:40:42PM +0200, Marco Davids via NANOG wrote:
> > We currently live in times where is actually fun to go IPv6-only. In my
> > case, as in: running a FreeBSD kernel compiled without the IPv4-stack.
> 
> Indeed, this is fun experimentation. Shaking the (source code) trees
> through excercises like these is a valuable way to identify gaps.
> 
> > It turns out that there underlying CDN's with domain names such as
> > ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
> > reside on authoritative name servers that *only* have an IPv4 address.
> 
> As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
> Fastly is working hard with select customers and friends to support IPv6
> for everyone.
> 
> ** SNIP **
> 
> as BGP traffic engineering) might be reluctant to offer IPv6 services
> "as if they are the same as IPv4". More study is required.
> 
> Tl;DR - work in progress! :-)

Some of the other CDNs do have IPv6 on the authorities and
should work without issues.

eg:

dig -6 +trace www.akamai.com.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: Redeploying most of 127/8, 0/8, 240/4 and *.0 as unicast

2021-11-19 Thread Jared Mauch



> On Nov 18, 2021, at 4:31 PM, Randy Bush  wrote:
> 
> as a measurement kinda person, i wonder if anyone has looked at how much
> progress has been made on getting hard coded dependencies on D, E, 127,
> ... out of the firmware in all networked devices.


At least the E space is largely usable last I checked (eg: IOS-XR, JunOS).  You 
can even measure traceroutes that have the class-e space in them from some 
cloud providers if your network and the intermediaries don’t do u-rpf as well.

Most OS’es (MacOS, Linux, various *BSD) also permit this as well.  My 
understanding is that the RedmondOS may not handle this, but if you need to 
number your infrastructure these addresses may work well.

- Jared

Re: Redeploying most of 127/8, 0/8, 240/4 and *.0 as unicast

2021-11-25 Thread Jared Mauch
On Fri, Nov 19, 2021 at 09:43:26AM -0800, Michael Thomas wrote:
> 
> On 11/19/21 8:27 AM, Randy Bush wrote:
> > these measurements would be great if there could be a full research-
> > style paper, with methodology artifacts, and reproducible results.
> > otherwise it disappears in the gossip stream of mailimg lists.
> > 
> Maybe an experimental rfc making it a rfc 1918-like subnet and implementing
> it on openwrt or something like that to see what happens. how many ip
> cameras and the like roll over and die? same for class E addresses too, I
> suppose. The question with anything that asks about legacy is how long the
> long tail actually is.
> 
> Mike, not that have any position on whether this is a good idea or not

I can tell you it's observable out there and if i use my home network
to follow default i can tell it is working through those devices at
least.

I agree with Randy it would be good if someone did this, it shouldn't be
too hard with ripe atlas and a provider deciding to announce something
like 240.2.3.0/24 to see if it can be reached.

That's at least a decent measurement and report, but the client
side OS will still be a variable that is difficult to digest.  Not sure
how many people are running very old IP stacks.  This is another hard to
measure problem.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: private 5G networks?

2021-12-06 Thread Jared Mauch



> On Dec 5, 2021, at 3:42 AM, Eliot Lear  wrote:
> 
> 
> On 01.12.21 15:17, Tom Beecher wrote:
>> While you are correct that it's just as illegal to intentionally interfere 
>> with the unlicensed wifi bands as it is with CBRS, the difference is that 
>> the FCC and regulatory bodies are much more likely to investigate and take 
>> action against intentional interference in these frequency ranges than they 
>> would be in the unlicensed wifi bands.
> 
> And there's a practical reason for that: establishing proof of unauthorized 
> use of a frequency is a heck of a lot easier than intentional interference.  
> All the former requires is triangulation of the offending station.  The 
> latter requires that plus a finding of intent. It CAN happen; but more often 
> than not what is actually found is a faulty piece of equipment that is 
> emitting something and everything else catching a bad harmonic. There was a 
> famous case about this in Wales in which an old television set took out a 
> town.[1]
> 
> Eliot
> 
> https://www.openreach.com/news/second-hand-tv-wipes-out-broadband-for-entire-village/
> 
> 

There was one in Oregon where it was transmitting on one of the ELB frequencies 
as well

https://www.nytimes.com/2004/11/01/technology/the-tv-that-sent-out-a-cry-for-help-via-satellite.html

- Jared

Re: Log4j mitigation

2021-12-11 Thread Jared Mauch
This is largely a patching exercise for people that use the software. If you 
use it, please patch. 

Sent via RFC1925 complaint device

> On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth  wrote:
> 
> The intricacies of Java are over my head, but I’ve been reading about this 
> Log4j issue that sounds pretty bad.
> 
> What do we know about this? What, if anything, can a network operator do to 
> help mitigate this? Or even an end user?
> 
> 
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> a...@andyring.com


Re: Log4j mitigation

2021-12-13 Thread Jared Mauch



> On Dec 13, 2021, at 2:24 PM, Owen DeLong  wrote:
> 
> The bigger problem seems to be the ever growing list of products you may be 
> using which depend on it potentially without your knowledge.

This isn’t a new problem.

This is an great modern example showing how deeply embedded things could be, 
and they get worse with each of these nesting technologies as well, it may be 
embedded in a docker or VM image, or the class could be in some other JAR or 
zip you are not aware of, or could come back with an overlapping class 
definition based on the order things get loaded.

The same was always true with shared libraries and too-generic function names.

It’s such a blast from the past as I had felt we had moved past many of these 
interpreted environment or parser things by properly encoding strings with a 
function.

I’m really amazed at how widespread this is and what enterprise applications 
have had to get patched due to them embedding this software.

- jared

Re: Can it really be this quiet?

2022-01-03 Thread Jared Mauch



> On Jan 3, 2022, at 2:53 PM, Job Snijders via NANOG  wrote:
> 
> Hi Allen,
> 
> Yes, it can be this quiet. It’s good news, it means the thing is mostly 
> working :-)
> 
> I wish everyone a happy and calm 2022!
> 

I’m surprised nobody talked about the major provider outages that happened over 
the break, there were several of them, but mostly in Asia.

- jared




Re: home router battery backup

2022-01-13 Thread Jared Mauch



> On Jan 13, 2022, at 12:28 PM, Chris Adams  wrote:
> 
> Once upon a time, Brandon Martin  said:
>> AT&T and Comcast don't seem to provide battery by default if you buy
>> voice service from them.
> 
> The only major power outage I've experienced at my house (I've been here
> over 20 years) was the May 2011 tornado outbreak, when TVA lost hundreds
> of distribution towers, and my local utility lost all feeds.  At the
> time, I had AT&T POTS, Comcast cable/Internet, and T-Mobile cell.

You are lucky.  In my areas we have many power outages and in my ~20 years in 
my current house, we have had several outages that went past 3 days in various 
weather (spring, summer, fall, winter)

Not only were we impacted by the NE blackout, but just in the short time I’ve 
had a generator it’s run around 0.5% of the time due to grid outage, with the 
prior year we had 7 different outages.

I also don’t believe the reporting is accurate from this source I’m doing a 
quick SWAG of, I think it’s been longer.  It’s so bad I monitor the grid 
voltage and have it in influxdb+grafana dashboard

(I recommend iotawatt if you want a neat device)

- Jared

Re: Anyone have contacts for Akamai GeoIP

2022-03-02 Thread Jared Mauch
Yup.. ping me with the details off-list

- Jared

> On Mar 2, 2022, at 7:02 PM, Christopher Munz-Michielin 
>  wrote:
> 
> Hey All,
> 
> Hoping someone has a contact at Akamai who can assist.
> 
> As part of my day job I run a DNS network and we've been having issues with 
> Akamai mis-locating the geolocation of some of our revolvers.  The most 
> egregious example is our resolver in Frankfurt being classified as 
> Australian, but there are some other instances as well.
> 
> This is an issue because we run fully recursive revolvers, so when a customer 
> queries our DNS server, we attempt to resolve the domain directly against the 
> authoritative name servers (Akamai in this case).  Because Akamai has 
> mis-located our IPs, we get handed an IP in the wrong hemisphere and our 
> customers experience 200+ ms latency to sites that should be regional.
> 
> We've tried reaching out via a couple of channels but have not gotten 
> anything back, wondering if anyone on the list knows a contact address for 
> Akamai GeoIP we can submit corrections to.
> 
> Cheers,



Re: IP Multicast Persistent Duplicate Packet Issue

2022-04-06 Thread Jared Mauch
If this is ASM, what device is the RP?  You may want to configure MSDP between 
PE1/PE2 to help if that’s the case, or is this SSM or something else you may 
want to just flip the PIM priority to make it pick what you want and see if you 
can tie it to your HSRP (Cisco, might I suggest VRRP so you could do 
dual-vendor later?) state, perhaps with EEM script to keep the config in sync 
if required.

- Jared

> On Apr 6, 2022, at 3:25 AM, Roman Islam  wrote:
> 
> Hi Everyone,
> 
> Has anyone experienced a TETRA Radio application issue if underlying IP 
> multicast transport sends persistent duplicate packets?
> 
> Here is my scenario as below:
> 
> PIM is running on the MPLS L3 VPN environment. C multicast is running on a 
> single VRF (TETRA) only. Source is running behind a dual home PE. HSRP, PIM 
> DR path is via PE1 to the source. Anycast RP is configured with PE1 & PE2. 
> 
> On the receiver side there is a single PE. When I check (S,G) route on the 
> receiver side PE default MDT is working as expected. After the threshold 
> exceeds it switches to the data MDT. PIM Assert mechanism winner is PE2 
> though (Since PE 2 has the highest IP and source is behind dual home PE with 
> equal cost I guess).
> 
> Can this be a reason for persistent duplicate multicast packets at the 
> receiver side; since the assert winner is PE2 but HSRP, PIM DR path is via 
> PE1 to the source? 
> 
> If this is the case is there any way to manually configure the assert winner 
> to PE1? 
> 
> Roman



Re: IP Multicast Persistent Duplicate Packet Issue

2022-04-06 Thread Jared Mauch
You can likely set "ip pim dr-priority" to get what you want

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipmulti/command/imc-cr-book/imc_i3.html#wp1384657000

- Jared

On Thu, Apr 07, 2022 at 09:07:34AM +1000, Roman Islam wrote:
> Thanks Jared. Yes it is SSM. I will try to flip PIM priority and align HSRP
> active.
> 
> My initial thought was keeping everything as it is. Can I simply flip PIM
> assert winner to PE1. Same way we can manually select STP root. Looks like
> this is probably not an option. Didn't find relevant doc on how to
> manipulate PIM assert winner in Cisco boxes though. Highest IP is the
> winner if IGP costs are the same to the source.
> 
> R
> 
> 
> On Thu, Apr 7, 2022 at 3:02 AM Jared Mauch  wrote:
> 
> > If this is ASM, what device is the RP?  You may want to configure MSDP
> > between PE1/PE2 to help if that’s the case, or is this SSM or something
> > else you may want to just flip the PIM priority to make it pick what you
> > want and see if you can tie it to your HSRP (Cisco, might I suggest VRRP so
> > you could do dual-vendor later?) state, perhaps with EEM script to keep the
> > config in sync if required.
> >
> > - Jared
> >
> > > On Apr 6, 2022, at 3:25 AM, Roman Islam  wrote:
> > >
> > > Hi Everyone,
> > >
> > > Has anyone experienced a TETRA Radio application issue if underlying IP
> > multicast transport sends persistent duplicate packets?
> > >
> > > Here is my scenario as below:
> > >
> > > PIM is running on the MPLS L3 VPN environment. C multicast is running on
> > a single VRF (TETRA) only. Source is running behind a dual home PE. HSRP,
> > PIM DR path is via PE1 to the source. Anycast RP is configured with PE1 &
> > PE2.
> > >
> > > On the receiver side there is a single PE. When I check (S,G) route on
> > the receiver side PE default MDT is working as expected. After the
> > threshold exceeds it switches to the data MDT. PIM Assert mechanism winner
> > is PE2 though (Since PE 2 has the highest IP and source is behind dual home
> > PE with equal cost I guess).
> > >
> > > Can this be a reason for persistent duplicate multicast packets at the
> > receiver side; since the assert winner is PE2 but HSRP, PIM DR path is via
> > PE1 to the source?
> > >
> > > If this is the case is there any way to manually configure the assert
> > winner to PE1?
> > >
> > > Roman
> >
> >

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: BGP Javascript Map/Visualization

2022-05-26 Thread Jared Mauch
On Thu, May 26, 2022 at 10:43:24PM +, Adam Thompson wrote:
> Neat.  Any idea who to ask questions of, regarding the incorrectness of the 
> data?  I would have assumed Job, but he's long gone from NTT, is this 
> abandonware or maintained?  Anyone know?
> 

I know where it's hosted, and it was a bit amusing when I left
NTT that he gave me grief about the as2914.net domain registration ... 

I would love to see it updated, even if it's not from the 2914
vantage point, i think he left some docs about how it was done.

    - jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

2022-05-29 Thread Jared Mauch



> On May 26, 2022, at 9:31 AM, Livingood, Jason via NANOG  
> wrote:
> 
>> Latency is a limitation for things that are generally relatively low 
>> bandwidth (interactive audio, zoom, etc.).
>> Higher bandwidth won’t solve the latency problem
> 
> +1
> IMO as we enter the 'post-gigabit era', an extra 1 Gbps to the home will 
> matter less than 100 ms or 500 ms lower working latency (optimally sub-50 ms, 
> if not sub-25 ms). The past is exclusively speed-focused -- the future will 
> be speed + working latency + reliability/resiliency + consistency of QoE + 
> security/protection + WiFi LAN quality.

This is always cute when posted from the haves vs the have-nots.  I’m watching 
a lot of people who don’t want to take government money, or play along flail at 
all of this.  They see the internet as for e-mail vs some futuristic use-case.

A few realities:

1) material cost is overall small for a fiber network (Even with the 250% price 
increase in the past ~24 months in materials)
2) Labor is the killer (this also has inputs of diesel fuel costs as the trucks 
that move the stuff are all diesel) reflecting 80%+ of the direct hard costs
3) There’s a lot of variable soft costs in permitting, engineering (Drawings) 
and network design inputs.
4) Many electric utilities have poor quality poles and want to charge tenants 
to upgrade them when they’ve ignored them for decades
5) Several companies have zero incentive to improve the QOE of the end-user 
service

Of course speed, latency, reliability matter.

It’s possible to hit people with varying technologies, and when you stick to 
one, be it PON, HFC, xDSL + FTTx, the other inputs come into play, be it the 
spectrum reserved for RF overlay on PON and HFC or otherwise.

You’re also seeing carriers walk away from new developments if they can’t be 
the monopoly option there, so it’s quite interesting watching what happens with 
my FTTH hat on.

I would say, if you’re looking to build or expand your networks, focus on how 
you can get the fiber out there, there’s a lot of money available if you’re 
willing to take it.  It might mean taking the USF money and the obligations 
that go with that in reporting, compliance, etc.. but those costs don’t have to 
be onerous if you are mindful of how the programs work and have the right 
integration/reporting.

- Jared

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

2022-05-29 Thread Jared Mauch
Sadly thus us repeating the same problematic data based on average usage by 
older Americans vs usage by younger people or those of us with several 
children. 

I agree with the average utilization but when it comes to those peaks my 
customers can finish their uploads or restores quickly when they do need it. If 
they are behind a limiter at 25m suddenly that FedEx or carrier pigeon seems 
best. 

Business I was at today says they need 40mbps 

- Jared 

Sent via RFC1925 complaint device

> On May 28, 2022, at 4:22 PM, Mike Hammett via NANOG  wrote:
> 
> 
> Most households have no practical use for more than 25 megs. More is better, 
> but let's not just throw money into a fire because of a marketing machine.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> 
> Midwest-IX
> http://www.midwest-ix.com
> 
> From: "Aaron Wendel" 
> To: nanog@nanog.org
> Sent: Monday, May 23, 2022 1:49:13 PM
> Subject: Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers
> 
> The Fiber Broadband Association estimates that the average US household 
> will need more than a gig within 5 years.  Why not just jump it to a gig 
> or more?
> 
> 
> On 5/23/2022 1:40 PM, Sean Donelan wrote:
> >
> > https://www.fcc.gov/document/fcc-proposes-higher-speed-goals-small-rural-broadband-providers-0
> >  
> >
> >
> > The Federal Communications Commission voted [May 19, 2022] to seek 
> > comment on a proposal to provide additional universal service support 
> > to certain rural carriers in exchange for increasing deployment to 
> > more locations at higher speeds. The proposal would make changes to 
> > the Alternative Connect America Cost Model (A-CAM) program, with the 
> > goal of achieving widespread deployment of faster 100/20 Mbps 
> > broadband service throughout the rural areas served by rural carriers 
> > currently receiving A-CAM support.
> >
> 
> 


Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

2022-05-31 Thread Jared Mauch



> On May 31, 2022, at 8:41 AM, Livingood, Jason via NANOG  
> wrote:
> 
> All the large DOCSIS networks of which I am aware are in fact working on 
> changing their spectrum plan and physical layer to enable higher US speeds 
> and in some cases symmetric multi-gig services.


Yes, I’ve seen Comcast claim to offer 2G symmetric services in their 
applications for funding from state/local authorities to expand their networks 
to unserved or underserved areas.  I have no reason to disbelieve this claim.  
I’ve been talking to vendors about what’s going on for the next generation of 
FTTx/PON and it looks quite attractive at this point, I’m excited to see the 
latency drops and speeds go up for people once they’re off DSL or DOCSIS over 
time.

In my early days of research for my “hobby ISP” as I call it, I looked at 
getting older docsis systems as an option/alternative, and it seemed to be 
worthwhile as without a TV overlay, there were more options for speed.

The reality is today once you have the infrastructure in place, if you planned 
well you can easily upgrade with overlays.

- Jared

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

2022-06-02 Thread Jared Mauch
On Wed, Jun 01, 2022 at 08:49:13PM -0700, Seth Mattinen wrote:
> On 6/1/22 8:12 PM, Mitchell Tanenbaum via NANOG wrote:
> > Believe it or not, there is cable within 500 yards, but they won’t
> > extend it. (:
> 
> 
> 50 feet across the street from me on the east side of the road is AT&T FTTH
> territory. My side of the street is not. F the west side apparently.

This is common sadly.  I had fiber 1200' from my house that was
unused and there may be no record of it, etc.. so it's just not possible
to happen.  Same goes for areas that have long-haul fiber passing them
but can't get service.

Not everyone is that lucky, but I've seen places with 2-3 fiber
providers that pass them and none offer service.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.


  1   2   3   4   5   6   7   8   9   10   >