Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

On 4/27/21 5:34 PM, Roy Smith wrote: I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/  and https://spi-tools-dev.toolforge.org/ .  Also: ssh: connect to host dev.tools.wmflabs.org

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

There's still a bit of upgrading happening but the recent network interruption should now be resolved. We're about to put Horizon back online so things should be back to normal shortly.  As always, some tools may have failed to recover from the network interruption so you're encouraged to check

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

Yep, it's related to the upgrade, working on it, thanks for your patience! Will update shortly. On 04/27 11:34, Roy Smith wrote: > I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ > and https://spi-tools-dev.toolforge.org/ >

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible. I have shut off the bastions to protect their local disks. Brooke Storm Staff SRE Wikimedia Cloud Services bst...@wikimedia.org >

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ . Also: ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable Could the VPS work be cau

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

Reminder: this is starting shortly. On 4/27/21 12:48 AM, Andrew Bogott wrote: We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes.  During the upgrade period Horizon will be disabled.  Ther