Juniper core dump issue, patch is on the way.
On Nov 7, 2011, at 11:41 AM, "Steve Dispensa" <dispe...@phonefactor.com> wrote: > Level 3 was down in KC, Chi, and San Jose (at least) for us between > about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too. > > -Steve > >> -----Original Message----- >> From: nanog-requ...@nanog.org [mailto:nanog-requ...@nanog.org] >> Sent: Monday, November 07, 2011 10:05 AM >> To: nanog@nanog.org >> Subject: NANOG Digest, Vol 46, Issue 15 >> >> Send NANOG mailing list submissions to >> nanog@nanog.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mailman.nanog.org/mailman/listinfo/nanog >> or, via email, send a message with subject or body 'help' to >> nanog-requ...@nanog.org >> >> You can reach the person managing the list at >> nanog-ow...@nanog.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of NANOG digest..." >> >> >> Today's Topics: >> >> 1. Re: Time Warner Telecom problems (Jon Lewis) >> 2. Re: Performance Issues - PTR Records (Leigh Porter) >> 3. Re: Time Warner Telecom problems (Ray Van Dolson) >> 4. General Internet Instability (Jared Mauch) >> 5. Re: TATA problems? (Pierre-Yves Maunier) >> 6. Re: TATA problems? (Leigh Porter) >> 7. Re: Time Warner Telecom problems (Joe Greco) >> 8. Re: TATA problems? (Kelly Kane) >> 9. Re: Time Warner Telecom problems (Blake Hudson) >> 10. RE: Time Warner Telecom problems (Thomas York) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST) >> From: Jon Lewis <jle...@lewis.org> >> To: Peter Pauly <ppa...@gmail.com> >> Cc: nanog@nanog.org >> Subject: Re: Time Warner Telecom problems >> Message-ID: <pine.lnx.4.61.1111071005540.24...@soloth.lewis.org> >> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed >> >> On Mon, 7 Nov 2011, Peter Pauly wrote: >> >>> Gizmodo is reporting problems at Time Warner Telecom .... we're >> suffering >>> from it too and calls to the NOC have not been answered so far... > does >>> anyone have any further information? >>> >>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us >> >> I noticed just a little while ago that we're having a lot of DNS fail. >> Initial findings were that several of the root-servers we were trying > to >> reach via our TWTelecom link were unreachable after 2 hops into TWT. >> >> 4 64-128-130-233.static.twtelecom.NET (64.128.130.233) 2.399 ms > 2.298 >> ms 2.338 ms >> 5 mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18) 11.571 ms >> 11.552 ms 9.467 ms >> 6 * * * >> 7 * * * >> 8 * * * >> >> For instance, a.root-servers.net is pingable from a rackspace server, > but >> not from our network (unless I shut off TWT, at which point it is, but >> it's apparently not the same a.root-servers.net instance rackspace > sees). >> I assume this is one of the root-servers being anycast. >> >> Shutting off our BGP with TWT didn't appear to help (though the >> root-servers became reachable)...so I assume there's more going on > than >> just TWT routing fail. >> >> ---------------------------------------------------------------------- >> Jon Lewis, MCP :) | I route >> Senior Network Engineer | therefore you are >> Atlantic Net | >> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ >> >> >> >> ------------------------------ >> >> Message: 2 >> Date: Mon, 7 Nov 2011 15:29:30 +0000 >> From: Leigh Porter <leigh.por...@ukbroadband.com> >> To: Bj?rn Mork <bj...@mork.no> >> Cc: "nanog@nanog.org" <nanog@nanog.org> >> Subject: Re: Performance Issues - PTR Records >> Message-ID: <53a4963f-4969-4a60-bf06-e690c7324...@ukbroadband.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> >> On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bj...@mork.no> wrote: >> >>> Leigh Porter <leigh.por...@ukbroadband.com> writes: >>> >>>> Indeed, there is no way I would allow that either. But really, >>>> providing a reverse zone and forward zone to match is a case of > five >>>> minutes and a shell script or a DNS that as Steinar said, will >>>> synthesise results. >>>> >>>> It's really not all that difficult.. >>> >>> No, not at all. It's just totally pointless. Any IPv6 address is > just >>> as pretty as a synthesized name. Maybe even prettier. Do you prefer >>> "2001:db8:1::2" or > "20010db8000100000000000000000002.rev.example.com"? >>> >>> If we're going to provide any reverse DNS for end users, then it is >>> because we can create names which actually improves something. >>> >>> >>> Bj?rn >>> >>> >> >> Yup it is pointless.. Mine are all ipadrress.domain which is of > course, >> pointless.. I suppose at least somebody would glean that perhaps its a >> home user rather than a business or server on that address but that's > all. >> >> With IPv6 arguably even more pointless as you say. >> >> ______________________________________________________________________ >> This email has been scanned by the MessageLabs Email Security System. >> For more information please visit http://www.messagelabs.com/email >> ______________________________________________________________________ >> >> >> >> ------------------------------ >> >> Message: 3 >> Date: Mon, 7 Nov 2011 07:28:18 -0800 >> From: Ray Van Dolson <rvandol...@esri.com> >> To: nanog@nanog.org >> Subject: Re: Time Warner Telecom problems >> Message-ID: <20111107152817.ga29...@esri.com> >> Content-Type: text/plain; charset=us-ascii >> >> On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote: >>> Gizmodo is reporting problems at Time Warner Telecom .... we're >> suffering >>> from it too and calls to the NOC have not been answered so far... > does >>> anyone have any further information? >>> >>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us >> >> FWIW, my home TWC connection dropped this morning for about 15 minutes >> (Southern California around 6:30AM'ish). Still could ping the default >> gateway, but packets weren't traversing much beyond that. >> >> Didn't investigate further, just headed into work. >> >> Ray >> >> >> >> ------------------------------ >> >> Message: 4 >> Date: Mon, 7 Nov 2011 10:31:31 -0500 >> From: Jared Mauch <ja...@puck.nether.net> >> To: Tom Hill <t...@ninjabadger.net> >> Cc: nanog@nanog.org >> Subject: General Internet Instability >> Message-ID: <b6567bc5-75e9-4e58-afa9-5adc0c2a7...@puck.nether.net> >> Content-Type: text/plain; charset=us-ascii >> >> On Nov 7, 2011, at 10:08 AM, Tom Hill wrote: >> >>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: >>>> We seem to be having some problems with our tata links - first seen > in >> EU >>>> about 45 minutes ago, now we're seeing problems in NA. I'm focused > on >> DNS, >>>> so I'm seeing a lot of timeouts/servfails, but our networking folks > are >>>> talking about links dropping. >>>> >>>> Anyone else seeing oddness on the NA Internet right now? >>>> >>>> http://downrightnow.com/ confirms - something is up. >>> >>> There are widespread issues across the Internet; certain versions of >>> Juniper firmware have core dumped after seeing a particular BGP > 'UPDATE' >>> message. >>> >>> (That's the running theory at least). >>> >>> It's affected multiple service providers, globally, not just those >>> connected to TATA. >> >> >> Pretty much any major BGP event will impact multiple providers. >> >> A threshold you should use to view the general instability (which I > find >> valuable, you may as well) is route views data. >> >> If you look at the BGP UPDATES archive sizes, you can see when > something >> happens, e.g.: >> >> http://archive.routeviews.org/bgpdata/2011.11/UPDATES/ >> >> Take a look at the size of the updates.20111107.1400.bz2 file and the > 1415 >> file. They are abnormally large compared to a normal period of time. >> This shows there were a lot of updates out there being processed and a >> reference to levels of instability. >> >> If you are not feeding route views or similar community projects, > please >> consider doing so. It helps paint the view for those doing analysis. >> >> - Jared >> >> >> ------------------------------ >> >> Message: 5 >> Date: Mon, 7 Nov 2011 16:33:15 +0100 >> From: Pierre-Yves Maunier <na...@maunier.org> >> To: Tom Hill <t...@ninjabadger.net> >> Cc: nanog@nanog.org >> Subject: Re: TATA problems? >> Message-ID: >> > <CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npnqzslctxuxeks9q...@mail.gmail.com> >> Content-Type: text/plain; charset=ISO-8859-1 >> >> 2011/11/7 Tom Hill <t...@ninjabadger.net> >> >>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: >>>> We seem to be having some problems with our tata links - first > seen in >> EU >>>> about 45 minutes ago, now we're seeing problems in NA. I'm > focused on >>> DNS, >>>> so I'm seeing a lot of timeouts/servfails, but our networking > folks >> are >>>> talking about links dropping. >>>> >>>> Anyone else seeing oddness on the NA Internet right now? >>>> >>>> http://downrightnow.com/ confirms - something is up. >>> >>> There are widespread issues across the Internet; certain versions of >>> Juniper firmware have core dumped after seeing a particular BGP > 'UPDATE' >>> message. >>> >>> (That's the running theory at least). >>> >>> It's affected multiple service providers, globally, not just those >>> connected to TATA. >>> >>> Tom >>> >>> >>> >> On our side all our 10.3R2.11 core dumped which made all our > interfaces >> flapped. >> I've been told 10.4R1.9 is affected too. >> >> -- >> Pierre-Yves Maunier >> >> >> ------------------------------ >> >> Message: 6 >> Date: Mon, 7 Nov 2011 15:45:18 +0000 >> From: Leigh Porter <leigh.por...@ukbroadband.com> >> To: Pierre-Yves Maunier <na...@maunier.org> >> Cc: "nanog@nanog.org" <nanog@nanog.org> >> Subject: Re: TATA problems? >> Message-ID: <7994af08-0622-434f-974f-fc9269469...@ukbroadband.com> >> Content-Type: text/plain; charset="us-ascii" >> >> >> My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd >> seemed stable. >> >> -- >> Leigh >> >> >> On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <na...@maunier.org> > wrote: >> >>> 2011/11/7 Tom Hill <t...@ninjabadger.net> >>> >>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote: >>>>> We seem to be having some problems with our tata links - first > seen in >> EU >>>>> about 45 minutes ago, now we're seeing problems in NA. I'm > focused on >>>> DNS, >>>>> so I'm seeing a lot of timeouts/servfails, but our networking > folks >> are >>>>> talking about links dropping. >>>>> >>>>> Anyone else seeing oddness on the NA Internet right now? >>>>> >>>>> http://downrightnow.com/ confirms - something is up. >>>> >>>> There are widespread issues across the Internet; certain versions > of >>>> Juniper firmware have core dumped after seeing a particular BGP >> 'UPDATE' >>>> message. >>>> >>>> (That's the running theory at least). >>>> >>>> It's affected multiple service providers, globally, not just those >>>> connected to TATA. >>>> >>>> Tom >>>> >>>> >>>> >>> On our side all our 10.3R2.11 core dumped which made all our > interfaces >>> flapped. >>> I've been told 10.4R1.9 is affected too. >>> >>> -- >>> Pierre-Yves Maunier >>> >>> > ______________________________________________________________________ >>> This email has been scanned by the MessageLabs Email Security > System. >>> For more information please visit http://www.messagelabs.com/email >>> > ______________________________________________________________________ >> >> ______________________________________________________________________ >> This email has been scanned by the MessageLabs Email Security System. >> For more information please visit http://www.messagelabs.com/email >> ______________________________________________________________________ >> >> >> >> ------------------------------ >> >> Message: 7 >> Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST) >> From: Joe Greco <jgr...@ns.sol.net> >> To: ppa...@gmail.com (Peter Pauly) >> Cc: nanog@nanog.org >> Subject: Re: Time Warner Telecom problems >> Message-ID: <201111071554.pa7fsphb045...@aurora.sol.net> >> Content-Type: text/plain; charset=us-ascii >> >>> Gizmodo is reporting problems at Time Warner Telecom .... we're >> suffering >>> from it too and calls to the NOC have not been answered so far... > does >>> anyone have any further information? >>> >>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us >> >> Actually, it looks to me like they mean "Time Warner", because that's >> what they said. >> >> The company once known as "Time Warner Telecom" has always been a >> different entity, and hasn't been known as that in some time, now >> being called "twtelecom." Much of that company is what was once >> known as inc.net, a Milwaukee area provider of the '90's. >> >> Time Warner Cable appears to have experienced an implosion this > morning, >> being out of service for about 11 minutes. During that time, packets >> originating here in Milwaukee quickly died in Chicago; >> >> 1 76.46.192.1 8.320 ms 9.900 ms 7.974 ms >> 2 24.160.230.32 7.967 ms 5.975 ms 8.479 ms >> 3 24.160.229.132 8.471 ms 7.969 ms 10.991 ms >> 4 24.160.229.193 9.972 ms 9.973 ms >> 24.160.229.197 9.985 ms >> 5 * * * >> 6 * * * >> >> while packets destined for RR all seemed to be headed out to SJC, from >> what I can tell. >> >> ... JG >> -- >> Joe Greco - sol.net Network Services - Milwaukee, WI - > http://www.sol.net >> "We call it the 'one bite at the apple' rule. Give me one chance [and] >> then I >> won't contact you again." - Direct Marketing Ass'n position on e-mail >> spam(CNN) >> With 24 million small businesses in the US alone, that's way too many >> apples. >> >> >> >> ------------------------------ >> >> Message: 8 >> Date: Mon, 7 Nov 2011 07:55:33 -0800 >> From: Kelly Kane <ke...@hawknetworks.com> >> To: Tim Vollebregt <t...@interworx.nl> >> Cc: nanog@nanog.org >> Subject: Re: TATA problems? >> Message-ID: >> > <CAKfXD0zrH9TJkN=7doge1uaq8zxvszyvtjyflltbic6+umy...@mail.gmail.com> >> Content-Type: text/plain; charset=UTF-8 >> >> On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <t...@interworx.nl> wrote: >>> >>> On #IX there are rumours about Junos version 10.3R2.11 being core > dumped >> and >>> rebooted, which makes sense. >> >> Perhaps related to Juniper PSN-2011-08-327? Did the whole router >> reboot, or just the service module? >> >> We saw one TATA session, and one Abovenet session flap. >> >> Kelly >> >> >> >> ------------------------------ >> >> Message: 9 >> Date: Mon, 07 Nov 2011 10:02:13 -0600 >> From: Blake Hudson <bl...@ispn.net> >> To: nanog@nanog.org >> Subject: Re: Time Warner Telecom problems >> Message-ID: <4eb80105.8060...@ispn.net> >> Content-Type: text/plain; charset=UTF-8; format=flowed >> >> >> Joe Greco wrote the following on 11/7/2011 9:54 AM: >>>> Gizmodo is reporting problems at Time Warner Telecom .... we're >> suffering >>>> from it too and calls to the NOC have not been answered so far... > does >>>> anyone have any further information? >>>> >>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us >>> Actually, it looks to me like they mean "Time Warner", because > that's >>> what they said. >>> >>> The company once known as "Time Warner Telecom" has always been a >>> different entity, and hasn't been known as that in some time, now >>> being called "twtelecom." Much of that company is what was once >>> known as inc.net, a Milwaukee area provider of the '90's. >>> >>> Time Warner Cable appears to have experienced an implosion this > morning, >>> being out of service for about 11 minutes. During that time, > packets >>> originating here in Milwaukee quickly died in Chicago; >> >> Using the looking glass from TWtelecom, we saw 30-60min outage > (roughly >> 8:30AM to 9:30AM CST) between the Kansas City location and our own >> server room in Kansas City. Other TWtelecom locations appeared to be >> unaffected. Perhaps TWtelecom is served by Timewarner or shares >> equipment in KC. Either way, none of our KC customers who were served >> via TWtelecom or Timewarner were able to reach us. Packets would hit >> Level 3 Communications and die in either direction at the border > between >> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. >> http://lglass.twtelecom.net/ >> >> >> >> ------------------------------ >> >> Message: 10 >> Date: Mon, 7 Nov 2011 11:04:59 -0500 >> From: "Thomas York" <strate...@fuhell.com> >> To: "'Blake Hudson'" <bl...@ispn.net>, <nanog@nanog.org> >> Subject: RE: Time Warner Telecom problems >> Message-ID: <c23401cc9d67$046baea0$0d430be0$@fuhell.com> >> Content-Type: text/plain; charset="utf-8" >> >> FWIW, We saw issues here in Indianapolis between TWTC and L3 up until > a >> few minutes ago. >> >> --Thomas York >> >> -----Original Message----- >> From: Blake Hudson [mailto:bl...@ispn.net] >> Sent: Monday, November 07, 2011 11:02 AM >> To: nanog@nanog.org >> Subject: Re: Time Warner Telecom problems >> >> >> Joe Greco wrote the following on 11/7/2011 9:54 AM: >>>> Gizmodo is reporting problems at Time Warner Telecom .... we're >>>> suffering from it too and calls to the NOC have not been answered > so >>>> far... does anyone have any further information? >>>> >>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us >>> Actually, it looks to me like they mean "Time Warner", because > that's >>> what they said. >>> >>> The company once known as "Time Warner Telecom" has always been a >>> different entity, and hasn't been known as that in some time, now >>> being called "twtelecom." Much of that company is what was once > known >>> as inc.net, a Milwaukee area provider of the '90's. >>> >>> Time Warner Cable appears to have experienced an implosion this >>> morning, being out of service for about 11 minutes. During that > time, >>> packets originating here in Milwaukee quickly died in Chicago; >> >> Using the looking glass from TWtelecom, we saw 30-60min outage > (roughly >> 8:30AM to 9:30AM CST) between the Kansas City location and our own > server >> room in Kansas City. Other TWtelecom locations appeared to be > unaffected. >> Perhaps TWtelecom is served by Timewarner or shares equipment in KC. >> Either way, none of our KC customers who were served via TWtelecom or >> Timewarner were able to reach us. Packets would hit Level 3 > Communications >> and die in either direction at the border between >> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa. >> http://lglass.twtelecom.net/ >> >> >> >> >> >> End of NANOG Digest, Vol 46, Issue 15 >> ************************************* >
