Re: Time Warner outage?

2014-08-28 Thread Chris Garrett
I believe this is what is commonly referred to in our industry as a “resume generating event”. On Aug 27, 2014, at 8:24 AM, Alexandru Suciu wrote: > Can't wait to see the postmortem report on this one. > > On 8/27/2014 3:05 PM, Adam Greene wrote: >> Came back for us, too, at about 7am. >> >>

Re: Time Warner outage?

2014-08-28 Thread Stephen Satchell
This just keeps getting better and better: "Yahoo Logo Will be right back... Thank you for your patience. Our engineers are working quickly to resolve the issue." My upstream is Charter Business... On 08/28/2014 04:05 AM, Chris Garrett wrote: > I believe this is what is commonly referred to in

Re: Time Warner outage?

2014-08-28 Thread Chris Lane
I heard the following, It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address when the connected to the network between 5:30 and 7:00. The funny part is upper Managment heard about it on the today show On Thu, Aug 28, 2014 at 8:46 AM,

RE: Time Warner outage?

2014-08-28 Thread Naslund, Steve
Something sounds really unlikely about that. Lack of DHCP would not cause reachability problems except for the clients. The trace below looks like a transit connection that should be unaffected by DHCP. Looks more like a routing issue. Also sounds unlikely that one DHCP server would be cover

Re: Time Warner outage?

2014-08-28 Thread Faisal Imtiaz
hehe... We call them 'Character Builder Events'. Everyone gets to find out how much pressure they can take, beak new grounds, what they are made off, etc etc. 'Shit Storms' come in waves. typically these are not caused by singular events, but more like a 'Series of Unfortunate Events'.

Re: Time Warner outage?

2014-08-28 Thread Chris Garrett
I have never seen BGP drop peers because of a DHCP failure. I would love to see the breakdown on that. TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in at the time of the event. Can anyone confirm one way or the other? O

Re: Time Warner outage?

2014-08-28 Thread Warren Bailey
Sounds more likely than "we broke the dhcp server". Sent from my T-Mobile 4G LTE Device Original message From: Chris Garrett Date: 08/28/2014 6:39 AM (GMT-08:00) To: "Naslund, Steve" Cc: nanog@nanog.org Subject: Re: Time Warner outage? I have never seen BGP drop peers bec

Re: Time Warner outage?

2014-08-28 Thread Chris Lane
Agreed on DHCP, just passing along something i had heard about. With that said, why wouldn't the TW guys just post something oh right they did, blame the other guy "Level3" On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey < wbai...@satelliteintelligencegroup.com> wrote: > Sounds more likely

RE: Time Warner outage?

2014-08-28 Thread Naslund, Steve
I don’t buy that excuse either. If Level 3 is doing fiber maintenance on any route and takes down your entire network, then you have a pretty poor backbone design. It is not Level 3s fault if you design your network such that a single route loss causes a huge outage. Steven Naslund Chicago IL

RE: Verizon connectivity issues

2014-08-28 Thread Bacon, Ricky (RJ)
Hi Chris! @Ryan, XO was accurate, the peering connection between 6 and 7 is not congested, nor is it taking any sort of errors. I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the co

RE: Time Warner outage?

2014-08-28 Thread Warren Bailey
As others have said, it's difficult to rationalize a situation where dhcp breaks and your existing routing breaks as well. Especially one that kills the Internet for a couple 100k subs. I'd shoot flames, but all too often I've been the guy working something like this after some maintenance becam

Re: Verizon connectivity issues

2014-08-28 Thread Christopher Morrow
On Thu, Aug 28, 2014 at 10:18 AM, Bacon, Ricky (RJ) wrote: > Hi Chris! howdy! > @Ryan, XO was accurate, the peering connection between 6 and 7 is not I had thought Ryan wasn't at XO but an XO customer... I could be mistaken though. congested, nor is it taking any sort of errors. > > I took a

Re: Time Warner outage?

2014-08-28 Thread George, Wes
I am not cleared to give further details, but in the hopes of providing a little more accurate info, I can point you at the following blog post http://www.twcableuntangled.com/2014/08/twc-identifies-cause-of-internet-ou tage/ Thanks, Wes George Anything below this line has been added by my com

Re: Time Warner outage?

2014-08-28 Thread Jay Ashworth
- Original Message - > From: "Steve Naslund" > I don’t buy that excuse either. If Level 3 is doing fiber maintenance > on any route and takes down your entire network, then you have a > pretty poor backbone design. It is not Level 3s fault if you design > your network such that a single r

Re: Time Warner outage?

2014-08-28 Thread Chris Garrett
Based on the link that Wes shared, sounds like someone made a BGP boo boo. Not that I routed damn near the whole internet down a customers DS3 or anything once, but I can see how it could happen and then propagate out of control before it was caught. On Aug 28, 2014, at 11:57 AM, Jay Ashworth

Contact at NetSuite DNS operations?

2014-08-28 Thread Lyle Giese
I discovered that NetSuite's dns servers ens0 & 1 .netsuite.com are invisible from Comcast's business services in the Chicago area(limits of my testing abilities) but I can reach these same servers from a Linode virtual system in the Dallas, TX area. Looks like it's been this way for at least

Re: Broken IPv6 firmware on U-Verse 3801HGV

2014-08-28 Thread Brandon Ewing
On Tue, Aug 26, 2014 at 01:50:29AM +, Ivan Kozik wrote: > All for naught, though. With IPv6 enabled, the 3801HGV crashed and > rebooted about once an hour. After unchecking "IPv6 LAN Enabled" in > http://192.168.1.254/xslt?PAGE=C_2_6 everything went back to normal. > This happened to me as

Re: Broken IPv6 firmware on U-Verse 3801HGV

2014-08-28 Thread Andrew Kirch
please refer to the e-mail I sent to this list last December about IPv6 and u-verse. On 8/28/2014 4:07 PM, Brandon Ewing wrote: On Tue, Aug 26, 2014 at 01:50:29AM +, Ivan Kozik wrote: All for naught, though. With IPv6 enabled, the 3801HGV crashed and rebooted about once an hour. After

Prefix hijacking, how to prevent and fix currently

2014-08-28 Thread Tarun Dua
AS Number 43239 AS Name SPETSENERGO-AS SpetsEnergo Ltd. Has started hijacking our IPv4 prefix, while this prefix was NOT in production, it worries us that it was this easy for someone to hijack it. http://bgp.he.net/AS43239#_prefixes 103.20.212.0/22 <- This belongs to us. 103.238.232.0/22 KNS T

Re: Time Warner outage?

2014-08-28 Thread Eric Stoltz
Looks like TW is having further issues this afternoon. Eric Stoltz Neovera On 08/28/2014 01:37 PM, Chris Garrett wrote: Based on the link that Wes shared, sounds like someone made a BGP boo boo. Not that I routed damn near the whole internet down a customers DS3 or anything once, but I can s

Re: Prefix hijacking, how to prevent and fix currently

2014-08-28 Thread Faisal Imtiaz
I would say, start off with the NOC Contact http://bgp.he.net/AS43239#_whois It is most likely that someone has fat-fingered the IP space.. Faisal Imtiaz Snappy Internet & Telecom - Original Message - > From: "Tarun Dua" > To: nanog@nanog.org > Sent: Thursday, August 28, 2014 12:55:25

Re: Prefix hijacking, how to prevent and fix currently

2014-08-28 Thread Fred Baker (fred)
On Aug 28, 2014, at 9:55 AM, Tarun Dua wrote: > AS Number 43239 > AS Name SPETSENERGO-AS SpetsEnergo Ltd. > > Has started hijacking our IPv4 prefix, while this prefix was NOT in > production, it worries us that it was this easy for someone to hijack > it. > > http://bgp.he.net/AS43239#_prefixe

Re: Prefix hijacking, how to prevent and fix currently

2014-08-28 Thread Suresh Ramasubramanian
In this case.. Google that ASN name. Then go upstream. On 29-Aug-2014 8:56 am, "Faisal Imtiaz" wrote: > I would say, start off with the NOC Contact > > http://bgp.he.net/AS43239#_whois > > It is most likely that someone has fat-fingered the IP space.. > > > Faisal Imtiaz > Snappy Internet & Tele

Re: Prefix hijacking, how to prevent and fix currently

2014-08-28 Thread Mark Andrews
See "whois -r AS43239". The long term solution is to deploy RPKI and only use transits which use RPKI. No RPKI support => no business. Additionally make RPKI a peering requirement. Mark In message , Tarun Dua writes: > AS Number 43239 > AS Name SPETSEN