Hi everybody,
As announced previously, I am *cutting off access to* *mwmaint servers
now. *Your homedirs have been copied over one last time.
Replacements are:
- deployment.eqiad.wmnet
- MediaWiki Periodic Jobs:
https://wikitech.wikimedia.org/wiki/Mw-cron_jobs
- MediaWiki Maintenance Scripts:
https://wikitech.wikimedia.org/wiki/Maintenance_scripts
- ldap-maint1001.eqiad.wmnet
- LDAP Maintenance:
https://wikitech.wikimedia.org/wiki/SRE/LDAP#Common_LDAP_administrative_actions
Please do not hesitate to reach out to me or a member of the SRE ServiceOps
team, in this email thread or on task
<https://phabricator.wikimedia.org/T397017>, if you have any question, or
use cases you feel have not been properly replicated.
Thank you,
On Wed, Jun 25, 2025 at 4:16 PM Clément Goubert <[email protected]>
wrote:
> Hi everybody,
>
> As MediaWiki periodic jobs (
> https://wikitech.wikimedia.org/wiki/Mw-cron_jobs) and all maintenance
> scripts use cases have been migrated off the maintenance servers, we are
> going to start the process of decommissioning them.
>
> The unsupported use cases for mwscript-k8s can be worked around by *using
> the old mwscript wrapper from the deployment server *(as described in
> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported
> ).
>
> *We have copied your home directories from mwmaint{1002,2002} to
> deploy1003 in the /srv/homedirs/mwmaint{1002,2002} directories*, so you
> can either copy these files to your homedir there, or clean them up if they
> are not useful anymore (please clean them up if at all possible! :D).
>
> We will soon proceed with *limiting ssh access to members of the ops
> group* as a soft decommission, then we will move on to actually
> decommissioning the servers.
>
> You can follow this process in https://phabricator.wikimedia.org/T397017
>
> Thank you,
>
> --
> Clément 'claime' Goubert (they/them)
> Senior SRE - Service Ops
> Wikimedia Foundation
>
> On Thu, Jun 5, 2025 at 5:55 PM Scott French <[email protected]>
> wrote:
>
>> Hi all -
>>
>> Following up on Reuven's message on Monday, 5th of May.
>>
>> *As of today, maintenance scripts can no longer run on the mwmaint
>> hosts, in advance of their decommissioning*. This follows the earlier
>> "hard" deprecation that began on the 12th of May.
>>
>> While the majority of maintenance script use cases have moved to
>> Kubernetes, some are not yet supported (e.g., sql.php [0]) or involve
>> additional complexity in that environment (e.g., large output files [1]).
>>
>> For these cases, as Reuven noted in his message on the 5th of May, users
>> are directed to the active deployment host (i.e., available at
>> deployment.eqiad.wmnet) to run their scripts on an interim basis.
>>
>> This interim solution will be phased out as support lands for the
>> remaining use cases.
>>
>> Many thanks,
>> Scott French
>> Service Ops SRE
>>
>> [0]
>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported
>> [1]
>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Output_to_a_file
>>
>> On Tue, May 6, 2025 at 1:40 PM Brooke Vibber <[email protected]>
>> wrote:
>>
>>> On Tue, May 6, 2025 at 1:17 PM Reuven Lazarus <[email protected]>
>>> wrote:
>>>
>>>> Foreachwiki and friends are available in the Kubernetes container now,
>>>> with a flag on the way to make it easy to invoke them via mwscript-k8s. (
>>>> https://phabricator.wikimedia.org/T378479) We've also resolved the
>>>> resource-consumption issues with the shell loop approach, so that option is
>>>> viable too.
>>>>
>>>
>>> Woohoo! Thanks! :D
>>>
>>>
>>>> (That is, simultaneously launching that number of jobs will no longer
>>>> overload the Kubernetes infrastructure itself; of course it's still
>>>> possible to use up the available compute resources, and it's also possible
>>>> -- as ever -- for aggressive maintenance scripts to overload shared
>>>> resources like the databases. But it's back on the table; consider it
>>>> remediated from the temporary guidance "don't do this, it's dangerous" to
>>>> the indefinite guidance "this is powerful, do it cautiously.")
>>>>
>>>
>>> *nods seriously*
>>>
>>>
>>>> Interactive scripts like eval.php can be used with mwscript-k8s in
>>>> --attach mode (which forwards stdin and stdout to your terminal). (
>>>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Input_on_stdin)
>>>> Separately, note there's also mw-debug-repl to get a shell in a mw-debug
>>>> pod, rather than a mw-script one. (
>>>> https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#MediaWiki_REPL
>>>> )
>>>>
>>>
>>> *success kid meme* yay! Eval lives. :D
>>>
>>>
>>>> That said, mwscript-k8s can't support {my,}sql.php specifically,
>>>> because we don't have a mysql client in the MW image and shouldn't add one.
>>>> For now, you can run `sql` from the deploy hosts instead of the mwmaint
>>>> hosts. The long-term replacement probably won't be a maintenance script at
>>>> all but a dbconfig-aware wrapper around a local mysql client. (
>>>> https://phabricator.wikimedia.org/T375910)
>>>>
>>>
>>> Awesome, that works fine for my needs.
>>>
>>> Thanks!
>>>
>>> -- brooke
>>> _______________________________________________
>>> Wikitech-l mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
>
>
--
Clément 'claime' Goubert (they/them)
Senior SRE
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
