Re: Level 3 voice outage

ay, October 4, 2016 1:09 PM > To: Marco Teixeira mailto:ad...@marcoteixeira.com>> > Cc: Shawn Ritchie mailto:shawnritc...@gmail.com>>; > NANOG list mailto:nanog@nanog.org>> > Subject: Re: Level 3 voice outage > > Possibly somebody YANGed when they should have yinged :) >

Re: Level 3 voice outage

..@gmail.com>>; NANOG list mailto:nanog@nanog.org>> Subject: Re: Level 3 voice outage Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>> wrote:

RE: Level 3 voice outage

] On Behalf Of Mel Beckman Sent: Tuesday, October 4, 2016 1:09 PM To: Marco Teixeira Cc: Shawn Ritchie ; NANOG list Subject: Re: Level 3 voice outage Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira mailto:ad...@marcoteixeira.com>&

Re: Level 3 voice outage

Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira mailto:ad...@marcoteixeira.com>> wrote: Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end

Re: Level 3 voice outage

Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There

Re: Level 3 voice outage

Just so. We are just speculating, but in light of recent GRE attack developments, the ddos possibility is worth considering. -mel beckman On Oct 4, 2016, at 12:54 PM, Shawn Ritchie mailto:shawnritc...@gmail.com>> wrote: Well, Level3 has by no means said that this was the result of a DDoS, tha

Re: Level 3 voice outage

... but back then they didn't have ITIL and the likes (at least not that i knew... but then again, i was just starting with networking back then), where a ChM process must produce risk, impact, etc, reports, and at least one peer review meeting for approval :) = Marco On Tue, Oct 4, 2016 at 8:33

Re: Level 3 voice outage

I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge.

Re: Level 3 voice outage

On Tue, 04 Oct 2016 20:10:44 +0100, Marco Teixeira said: > Had it been an approved chance it would have been > rolled back i guess... See the 1990 ATT long-distance collapse for a worked example. pgp_Sro2vzM5h.pgp Description: PGP signature

Re: Level 3 voice outage

765 Gbps per second directed at a router’s interface IP might give the router pause, so to speak :) -mel On Oct 4, 2016, at 12:10 PM, Marco Teixeira mailto:ad...@marcoteixeira.com>> wrote: Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had

Re: Level 3 voice outage

Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply... Em 04/10/2016 19:54, "Mel Beckman" escreveu: > Sure. The recent release of the I

Re: Level 3 voice outage

Sure. The recent release of the IoT DDoS attack code in the wild. -mel > On Oct 4, 2016, at 11:42 AM, valdis.kletni...@vt.edu wrote: > > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said: > >> This could be DoS attack. > > Or a missing comma in a code update. > > Or a fumble-fingered NOC

Re: Level 3 voice outage

On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said: > This could be DoS attack. Or a missing comma in a code update. Or a fumble-fingered NOC monkey. Or You have any reason to suspect a DoS attack rather than all the other possibilities? pgp03Ijae5BSG.pgp Description: PGP signature

Re: Level 3 voice outage

We are seeing high latencies and loss of transit through Level3 in Southern California. Level 3 has opened Global Ticket 115724: Network devices in multiple markets in North America are rebooting, impacting IP services.” This could be DoS attack. > On Oct 4, 2016, at 11:06 AM, Jason Hellenth

Re: Level 3 voice outage

Patience Obi Wan ! They are investigating the root cause and like most root causes they don’t just hold out a flag and say here I am !!! > On Oct 4, 2016, at 10:32, Ivaylo Katovski wrote: > > When will L3 notify their customers for the outage?!? According to l3 > twitter account their are aware

Re: Level 3 voice outage

When will L3 notify their customers for the outage?!? According to l3 twitter account their are aware of the voice impact and working on it On Oct 4, 2016 11:03 AM, "Mark Stevens" wrote: > Is anyone noticing issue with Level 3 voice? I can't even call their 800 > number using one of my other car

RE: [BULK] Re: Level 3 voice outage

Boise Idaho is affected... -Original Message- From: NANOG [mailto:nanog-bounces+ascher=slhs@nanog.org] On Behalf Of Benjamin Hatton Sent: Tuesday, October 04, 2016 9:04 AM To: nanog@nanog.org Subject: [BULK] Re: Level 3 voice outage Importance: High WARNING: This email

RE: Level 3 voice outage

There's all sorts of people online reporting problems with Level3 voice services - we're down for outbound calling, inbound is spotty. Their portal is unreachable, can't get to any of their #'s...seems like the world is on fire over there. We were a previous TW Telecom customer before Level3 b

Re: Level 3 voice outage

We can't make any LD calls through Level3 either. Thank you, Shane Heupel > On Oct 4, 2016, at 10:06 AM, Benjamin Hatton wrote: > > We just got a notification from one of our upstream Voice providers that > they are seeing problems both inbound and outbound that seems to be Level 3 > related >

Re: Level 3 voice outage

Major Level 3 outage; they are currently not taking support calls. On 10/4/16, 8:01 AM, "NANOG on behalf of Mark Stevens" wrote: >Is anyone noticing issue with Level 3 voice? I can't even call their 800 >number using one of my other carriers. > >Mark

Re: Level 3 voice outage

On Tue, 4 Oct 2016, Mark Stevens wrote: Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers. Yes, major outage: http://downdetector.com/status/level3 Customers all over the mid-atlantic region are down.

RE: Level 3 voice outage

We are back in action here in MA. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of timrutherf...@c4.net Sent: Tuesday, October 4, 2016 11:20 AM To: nanog@nanog.org Subject: RE: Level 3 voice outage Our upstream Voice provider is also having inbound issues with

Re: Level 3 voice outage

that were ported into their system, but the main trunk number is working. > > -Tim > > -Original Message- > From: NANOG [mailto:nanog-bounces+timrutherford=c4@nanog.org] On > Behalf Of Benjamin Hatton > Sent: Tuesday, October 4, 2016 11:04 AM > To: nanog@nanog

Re: Level 3 voice outage

Our provider is down as well. Saying Level 3 issues. Anyone know what's going on or and ETR? On Tue, Oct 4, 2016 at 10:06 AM, Kevin Kadow wrote: > Level 3 has issues nationwide today. We have voice outages at our offices > in multiple cities, Chicago and out west. > > On Tue, Oct 4, 2016 at 11:

RE: Level 3 voice outage

4, 2016 11:04 AM To: nanog@nanog.org Subject: Re: Level 3 voice outage We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000

Re: Level 3 voice outage

There's a long thread over at the outages list. Sign-up for the outages list is at https://puck.nether.net/mailman/listinfo/outages and the archive for the whole thread starts at https://puck.nether.net/pipermail/outages/2016-October/009609.html. Matt Freitag Network Engineer I Information Technol

Re: Level 3 voice outage

Level 3 has issues nationwide today. We have voice outages at our offices in multiple cities, Chicago and out west. On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens wrote: > Is anyone noticing issue with Level 3 voice? I can't even call their 800 > number using one of my other carriers. > > Mark >

Re: Level 3 voice outage

We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000 bhat...@htva.net www.htva.net On Tue, Oct 4, 2016 at 11:01 AM, Mark Steve

Re: Level 3 voice outage?

Ran across this earlier, it sounds bad. https://www.reddit.com/r/networking/comments/504xbo/level_3_voice_outage_global_ticket_being_worked/ Spencer Ryan | Senior Systems Administrator | sr...@arbor.net Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arborn