Not a famous operational issue, but in 2000, we had a major outage of
our dialup modem pool.
The owner of the building was re-skinning the outside using Styrofoam
and stucco. A bunch of the Styrofoam
had blocked the roof drains on the podium section of the building,
immediately above our equipment room.
A flash rainstorm filled the entire flat roof, and water came back in
over the flashings, and poured directly in
to our dialup modem pool through the hole in the concrete roof deck
where the drain pipe protruded through.
In retrospect, it was a monumentally stupid place to put our main
modem pool, but we didn't realize what was
above the drop ceiling - and that it was roof, not the other 11
floors of the building.
1 bay of 6 shelves of USR TC 1000 HiperDSPs were now very wet and
blinking funny patterns on their LEDs.
Fortunately, our vendor in Toronto (4 hour drive away) had stock of
equipment that another customer kept
delaying shipment on. They got their staff in, started un-boxing
and, slotting cards. We spent a few hours
tearing out the old gear and getting ready for replacements.
We left Windsor, Ontario at around 12:00am - same time they left
Toronto, heading towards us. We coordinated
a meet at one of the rural exits along Highway 401 at a closed gas
station at around 2am.
Everything was going so well until a cop pulled up, and asked us what
we were doing, as we were slinging
modem chassis between the back of the vendor's SUV and our van... We
calmly explained
what happened. He looked between us a couple of times, shook his
head and said "well, good luck with that",
got back in his car and drove away.
We had everything back online within 14 hours of the initial outage.
At 02:37 PM 16/02/2021, John Kristoff wrote:
Friends,
I'd like to start a thread about the most famous and widespread Internet
operational issues, outages or implementation incompatibilities you
have seen.
Which examples would make up your top three?
To get things started, I'd suggest the AS 7007 event is perhaps the
most notorious and likely to top many lists including mine. So if
that is one for you I'm asking for just two more.
I'm particularly interested in this as the first step in developing a
future NANOG session. I'd be particularly interested in any issues
that also identify key individuals that might still be around and
interested in participating in a retrospective. I already have someone
that is willing to talk about AS 7007, which shouldn't be hard to guess
who.
Thanks in advance for your suggestions,
John
--
Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4
tel. 519-985-8410
fax. 519-985-8409
