gmodena added a comment.
Hi, Thanks for reaching out. Roughly speaking, we start to throttle connections (for bots that respect maxlag) when the change propagation lag between wikidata.org and the WDQS secondary store is higher than 10 minute for a sustained amount of time. A typical reason for this lag increase is when WDQS is put under heavy read load, and writes can't keep up. WDQS has been under heavy load starting Thursday, April 2. This had a cascading effect on lag and bot throttling (https://grafana.wikimedia.org/goto/dfhtdl5py8qv4f?orgId=1). We have alerting and operational <https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook/High_replication_lag_and_query_timeout> processes <https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbooks/ElevatedMaxLagWDQS> in place to mitigate this issue. We've been tracking load since early alerts started to fire on Thursday. Unfortunately, we had several actors concurrently putting WDQS under strain for the past 5 days, that defied automated remediation we have in place and required manual intervention. This situation resulted in bot throttling kicking off more than we would have liked. We've seen spikes in load and lag also this morning CEST (https://grafana.wikimedia.org/goto/dfid0zv54dqm8e?orgId=1), as we keep monitoring the situation. TASK DETAIL https://phabricator.wikimedia.org/T421642 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
_______________________________________________ pywikibot-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
