Re: Austin Colo Offline

2016-11-15 Thread Andy Doan
The Austin Colo (and DeveloperCloud) should be back online now. Our main router had a fan that failed. Despite the other 3 working fine, I had to steal one from another server to get it running. I had to steal a fan from the server handling all serial console connections to r1-a* servers. So those

Re: Austin Colo Offline

2016-11-15 Thread Andy Doan
Its down again. Sorry for troubles. On 15 November 2016 at 10:02, Andy Doan wrote: > Everything should be back online now. The team is now going to comb > over logs to understand how this happened. > > On 14 November 2016 at 23:28, Andy Doan wrote: >> The Austin colo router has become unreachabl

Re: Austin Colo Offline

2016-11-15 Thread Andy Doan
Everything should be back online now. The team is now going to comb over logs to understand how this happened. On 14 November 2016 at 23:28, Andy Doan wrote: > The Austin colo router has become unreachable and everything behind it is > offline. I'll be heading there in my Tuesday morning to debug

Re: Austin colo offline?

2016-09-17 Thread Renato Golin
On 17 September 2016 at 19:16, Andy Doan wrote: > The collocation facility hosting our servers suffered a major HVAC > failure. To compound matters, their monitoring network failed to notice > and temperatures got pretty hot for about 3 rows of racks (of which we > were one). Ouch! Mr. Robot stri

Re: Austin colo offline?

2016-09-17 Thread Andy Doan
The collocation facility hosting our servers suffered a major HVAC failure. To compound matters, their monitoring network failed to notice and temperatures got pretty hot for about 3 rows of racks (of which we were one). At this point, I think everything is back online but please let me know if so