Re: [Cloud] Wiki replicas: Compilation of input and logged cross-DB queries

2021-04-27 Thread Huji Lee
Hi, Joaquin! I was looking at the page the other day. I agree with your summary above, except I would add one more prominent case which somehow did not appear in the random query analysis but I think is critical: - Querying CentralAuth and another DB An example is this query

[Cloud] Wiki replicas: Compilation of input and logged cross-DB queries

2021-04-27 Thread Joaquin Oltra Hernandez
Hi! Like I mentioned in the past, I've been cataloging input from developers and doing some analysis from the random query logger that Brooke coded in the old cluster to get a better idea of the kinds of cross DB queries being performed. Wiki Replicas Cross-DB Query Data

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Andrew Bogott
There's still a bit of upgrading happening but the recent network interruption should now be resolved. We're about to put Horizon back online so things should be back to normal shortly.  As always, some tools may have failed to recover from the network interruption so you're encouraged to check

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread David Caro
Yep, it's related to the upgrade, working on it, thanks for your patience! Will update shortly. On 04/27 11:34, Roy Smith wrote: > I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ > and https://spi-tools-dev.toolforge.org/ >

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Brooke Storm
Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible. I have shut off the bastions to protect their local disks. Brooke Storm Staff SRE Wikimedia Cloud Services bst...@wikimedia.org >

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Roy Smith
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ . Also: ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable Could the VPS work be cau

Re: [Cloud] [Cloud-announce] cloud-vps maintenance at 14:00 UTC

2021-04-27 Thread Andrew Bogott
Reminder: this is starting shortly. On 4/27/21 12:48 AM, Andrew Bogott wrote: We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes.  During the upgrade period Horizon will be disabled.  Ther