On Nov 30, 2011, at 9:51 AM, Blake Hudson wrote:
> Stefan wrote the following on 11/30/2011 8:53 AM: >> On Wed, Nov 30, 2011 at 8:21 AM, Brad Fleming<bdfle...@gmail.com> wrote: >>> On Nov 29, 2011, at 8:17 PM,<compt...@kc.rr.com> wrote: >>> >>>> We lost several of our GigE links to AT&T for 6 hours on 11/19, anyone >>>> else see this and get a root cause from AT&T? All I can get is that they >>>> believe a change caused the issue. >>>> >>> We lost several (but not all) of our Optiman circuits on 11/19 at about >>> 10:20am. We were told the root issue was that all VLANs in one of their >>> switches had been accidentally deleted / removed. We were never able to get >>> any additional detail (like "how") but services were restored about 16:45. >> +1 to the above - we received the following RFO, from the their NOC: >> >> "All impacted VLANS were rebuilt to restore service. It is believed >> there were some configuration changes that caused the VLAN troubles. A >> case has been opened with Cisco to further investigate the root >> cause." >> > > Sounds like a VTP mishap. > That was my first thought as well.. it would just surprise me if a huge provider like AT&T was using VTP instead of using a provisioning tool that automates the manual pruning process to avoid issues like this. In either case I'm a customer and will likely never be told what went wrong. I'm OK with that so long as it doesn't happen again!
