On 10/21/19 9:49 PM, Brooke Storm wrote:
> With a redundant power supply upgrade going on this week in the datacenter
> that
> could affect the VM that Toolsdb runs on, we anticipate a brief outage
> Thursday
> 10/24 @11am UTC of the mysql service to protect data in case anything goes
> wrong. Th
On 10/21/19 7:56 PM, Martin Urbanec wrote:
> Is there something you missed to say?
>
> "operation which is migrating data stored in Redis which can be tricky. The o"
That's a typo/leftover from me rewording that sentence.
Sorry for that :-)
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Serv
An entirely surprising side-effect of this maintenance is causing
chronic database instability. We're working to resolve this but in the
meantime the tools database server is likely to be up and down several
times. We'll update once things are stable again.
Sorry for the (ongoing) interrupti
Thanks to last-minute intervention by Jaime Crespo, toolsdb is back to
working as normal. Some context can be found at
https://phabricator.wikimedia.org/T236384
-Andrew + wmcs team
On 10/24/19 10:23 AM, Andrew Bogott wrote:
An entirely surprising side-effect of this maintenance is causing
c
Two-plus hours later I’m still seeing frequent DB errors.
Sent from my iPhone
> On Oct 24, 2019, at 9:04 AM, Andrew Bogott wrote:
>
> Thanks to last-minute intervention by Jaime Crespo, toolsdb is back to
> working as normal. Some context can be found at
> https://phabricator.wikimedia.org/
On Thu, Oct 24, 2019 at 12:24 PM Russell Blau wrote:
>
> Two-plus hours later I’m still seeing frequent DB errors.
You are very correct. The 'all clear' from Andrew seemed like the
right thing at the time, but we are still having stability issues with
the ToolsDB service.
We have an active track
