On 7/10/15 7:54 AM, Jürgen Jaritsch wrote: > Wow .... Level3 responded to me that they had an issue last night .... but > they simply did nothing ... for at least 10 hours they did nothing to fix the > issue:
It's more likely that there's a certain amount of nonsense and a lot of loose ends in the ticketing system; than that the issue resolved itself without intervention. You might reasonably conclude that the corrective action and the ticket are unrelated at least as far as the noc is concerned. > ### > > Event Case ID: 9446216 > Location: Los Angeles, CA > Impacted For: 10 hours 52 minutes > ETR: Unknown > Bridge: N/A > > > 08:52 GMT - Event Conclusion Summary > > Start Time: 07/09 19:48 GMT > Stop Time: 07/10 06:40 GMT > > Root Cause: A packet loss issue in Los Angeles, CA. > > Fix Action: The packet loss issue resolved before any action was taken. > Summary: The IP NOC is currently investigated a packet loss issue in Los > Angeles, CA that was impacting IP services. The packet loss issue resolved > before any action was taken and the IP NOC deemed services are stable after > monitoring was concluded. > > 07:53 GMT - The IP NOC is currently monitoring services after the packet loss > issue resolved before any action was taken. An update will be sent when > traffic is deemed stable or if status changes. > > 06:43 GMT - The IP NOC is currently investigating a packet loss issue in Los > Angeles, CA that is impacting IP services. Troubleshooting efforts are > ongoing with no estimated time to restore services available at this point. > Please be advised that updates for this event will be relayed hourly unless > otherwise noted. > > ### > > > best regards > > Jürgen Jaritsch > Head of Network & Infrastructure > > ANEXIA Internetdienstleistungs GmbH > > Telefon: +43-5-0556-300 > Telefax: +43-5-0556-500 > > E-Mail: j...@anexia.at > Web: http://www.anexia.at > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt > Geschäftsführer: Alexander Windbichler > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 > > > -----Ursprüngliche Nachricht----- > Von: Jürgen Jaritsch > Gesendet: Freitag, 10. Juli 2015 15:47 > An: 'Ca By'; Joseph Jenkins > Cc: nanog@nanog.org > Betreff: AW: Level3 routing issue US west coast > > Hi, > > sitting here and watching the packet loss coming and going :(. > > It changes every 10-25min. Looks like an massive issue in San Jose - routers > out there sometimes have an latency from 5-6 SECONDS ... > > best regards > > Jürgen Jaritsch > Head of Network & Infrastructure > > ANEXIA Internetdienstleistungs GmbH > > Telefon: +43-5-0556-300 > Telefax: +43-5-0556-500 > > E-Mail: j...@anexia.at > Web: http://www.anexia.at > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt > Geschäftsführer: Alexander Windbichler > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 > > > -----Ursprüngliche Nachricht----- > Von: NANOG [mailto:nanog-boun...@nanog.org] Im Auftrag von Ca By > Gesendet: Freitag, 10. Juli 2015 15:42 > An: Joseph Jenkins > Cc: nanog@nanog.org > Betreff: Re: Level3 routing issue US west coast > > On Friday, July 10, 2015, Joseph Jenkins <j...@breathe-underwater.com> wrote: > >> Level3 had an issue with one of their core routers in Los Angeles last >> night(7pm Pacific) and early this morning(1am Pacific). Last update to my >> trouble ticket had the issue still being reviewed by engineering, but that >> a core router was dropping packets. >> >> > I have seen this several times with level3. They confirm packets are > dropping and sevice is degraded yet they refuse to take tactical corrective > action for hours and hours. > > Makes me furious. > > CB > > >>> On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <j...@anexia.at >> <javascript:;>> wrote: >>> >>> Hi, >>> >>> does anyone else experience issues with the Level3 network at the US >> west coast? We see lots of broken paths like this: >>> >>> Packets >> Pings >>> Host Loss% Snt Last >> Avg Best Wrst StDev >>> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 >> 0.5 0.2 18.1 1.2 >>> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 >> 9.9 0.3 361.1 40.1 >>> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 >> 7.7 6.3 49.7 5.3 >>> 4. win-b4-link.telia.net 0.0% 230 6.6 >> 6.8 6.4 20.2 1.5 >>> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 >> 9.3 6.3 69.1 9.6 >>> 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 >> 165.0 164.5 194.9 2.6 >>> 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 >> 164.8 164.5 174.1 0.9 >>> 8. 4.53.208.102 34.3% 230 634.9 >> 310.7 168.5 680.1 199.7 >>> 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 >> 455.2 304.9 954.6 203.4 >>> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 >> 441.4 323.1 822.0 182.1 >>> 11. 211.115.201.92 38.4% 230 289.8 >> 412.9 289.6 846.7 185.4 >>> 12. 61.250.89.2 35.8% 230 290.6 >> 439.4 290.2 804.3 205.1 >>> 13. ??? >>> >>> Trace from NYC is also broken: >>> >>> Packets >> Pings >>> Host Loss% Snt >> Last Avg Best Wrst StDev >>> 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 >> 0.4 4.3 0.4 57.8 13.3 >>> 2. nyk-b5-link.telia.net 0.0% 30 >> 0.3 0.4 0.3 0.9 0.1 >>> 3. ??? >>> 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 >> 71.7 73.2 71.7 98.7 5.5 >>> 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 >> 71.8 71.8 71.7 72.0 0.1 >>> 6. 4.53.208.102 31.0% 30 >> 569.6 250.7 70.5 579.3 231.2 >>> 7. 63.218.250.73 31.0% 30 >> 672.6 355.5 178.0 672.6 232.1 >>> >>> >>> At 10:24 UTC+2 it was even more broken: >>> >>> Packets >> Pings >>> Host Loss% Snt Last >> Avg Best Wrst StDev >>> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 >> 0.5 0.3 39.7 2.2 >>> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 >> 6.7 0.3 198.1 26.3 >>> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 >> 7.6 6.4 43.6 4.5 >>> 4. win-b4-link.telia.net 0.0% 326 6.7 >> 7.4 6.3 43.1 3.2 >>> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 >> 9.2 6.3 73.2 10.1 >>> 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 >> 165.5 164.5 176.7 1.5 >>> ae-2-70.edge1.SanJose3.Level3.net >>> 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 >> 165.8 164.6 204.2 3.9 >>> ae-2-70.edge1.SanJose3.Level3.net >>> 8. 205.129.5.70 74.2% 326 799.9 >> 487.2 169.0 799.9 305.7 >>> 4.53.208.102 >>> 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. >> 701.0 308.7 3716. 510.6 >>> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 >> 643.0 323.4 960.4 307.6 >>> 11. 211.115.201.92 68.9% 326 925.3 >> 674.2 289.8 932.3 296.6 >>> 12. 61.250.89.2 72.9% 326 928.5 >> 637.2 291.9 928.5 304.3 >>> 13. ??? >>> >>> >>> best regards >>> >>> Jürgen Jaritsch >>> Head of Network & Infrastructure >>> >>> ANEXIA Internetdienstleistungs GmbH >>> >>> Telefon: +43-5-0556-300 >>> Telefax: +43-5-0556-500 >>> >>> E-Mail: j...@anexia.at <javascript:;><mailto:j...@anexia.at <javascript:;>> >>> Web: http://www.anexia.at<http://www.anexia.at/> >>> >>> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt >>> Geschäftsführer: Alexander Windbichler >>> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT >> U63216601 >>> >> >> >
signature.asc
Description: OpenPGP digital signature
