Hi everybody, *mwmaint servers will be decommed next week (July 28-July 31).*
If you still have concerns or data that was not copied over to deploy1003, please reach out to me or a member of the SRE ServiceOps team, in this email thread or on task <https://phabricator.wikimedia.org/T397017>. Thank you, On Wed, Jul 9, 2025 at 12:03 PM Clément Goubert <[email protected]> wrote: > Hi everybody, > > As announced previously, I am *cutting off access to* *mwmaint servers > now. *Your homedirs have been copied over one last time. > Replacements are: > > - deployment.eqiad.wmnet > - MediaWiki Periodic Jobs: > https://wikitech.wikimedia.org/wiki/Mw-cron_jobs > - MediaWiki Maintenance Scripts: > https://wikitech.wikimedia.org/wiki/Maintenance_scripts > - ldap-maint1001.eqiad.wmnet > - LDAP Maintenance: > > https://wikitech.wikimedia.org/wiki/SRE/LDAP#Common_LDAP_administrative_actions > > Please do not hesitate to reach out to me or a member of the SRE > ServiceOps team, in this email thread or on task > <https://phabricator.wikimedia.org/T397017>, if you have any question, or > use cases you feel have not been properly replicated. > > Thank you, > > > On Wed, Jun 25, 2025 at 4:16 PM Clément Goubert <[email protected]> > wrote: > >> Hi everybody, >> >> As MediaWiki periodic jobs ( >> https://wikitech.wikimedia.org/wiki/Mw-cron_jobs) and all maintenance >> scripts use cases have been migrated off the maintenance servers, we are >> going to start the process of decommissioning them. >> >> The unsupported use cases for mwscript-k8s can be worked around by *using >> the old mwscript wrapper from the deployment server *(as described in >> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported >> ). >> >> *We have copied your home directories from mwmaint{1002,2002} to >> deploy1003 in the /srv/homedirs/mwmaint{1002,2002} directories*, so you >> can either copy these files to your homedir there, or clean them up if they >> are not useful anymore (please clean them up if at all possible! :D). >> >> We will soon proceed with *limiting ssh access to members of the ops >> group* as a soft decommission, then we will move on to actually >> decommissioning the servers. >> >> You can follow this process in https://phabricator.wikimedia.org/T397017 >> >> Thank you, >> >> -- >> Clément 'claime' Goubert (they/them) >> Senior SRE - Service Ops >> Wikimedia Foundation >> >> On Thu, Jun 5, 2025 at 5:55 PM Scott French <[email protected]> >> wrote: >> >>> Hi all - >>> >>> Following up on Reuven's message on Monday, 5th of May. >>> >>> *As of today, maintenance scripts can no longer run on the mwmaint >>> hosts, in advance of their decommissioning*. This follows the earlier >>> "hard" deprecation that began on the 12th of May. >>> >>> While the majority of maintenance script use cases have moved to >>> Kubernetes, some are not yet supported (e.g., sql.php [0]) or involve >>> additional complexity in that environment (e.g., large output files [1]). >>> >>> For these cases, as Reuven noted in his message on the 5th of May, users >>> are directed to the active deployment host (i.e., available at >>> deployment.eqiad.wmnet) to run their scripts on an interim basis. >>> >>> This interim solution will be phased out as support lands for the >>> remaining use cases. >>> >>> Many thanks, >>> Scott French >>> Service Ops SRE >>> >>> [0] >>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported >>> [1] >>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Output_to_a_file >>> >>> On Tue, May 6, 2025 at 1:40 PM Brooke Vibber <[email protected]> >>> wrote: >>> >>>> On Tue, May 6, 2025 at 1:17 PM Reuven Lazarus <[email protected]> >>>> wrote: >>>> >>>>> Foreachwiki and friends are available in the Kubernetes container now, >>>>> with a flag on the way to make it easy to invoke them via mwscript-k8s. ( >>>>> https://phabricator.wikimedia.org/T378479) We've also resolved the >>>>> resource-consumption issues with the shell loop approach, so that option >>>>> is >>>>> viable too. >>>>> >>>> >>>> Woohoo! Thanks! :D >>>> >>>> >>>>> (That is, simultaneously launching that number of jobs will no longer >>>>> overload the Kubernetes infrastructure itself; of course it's still >>>>> possible to use up the available compute resources, and it's also possible >>>>> -- as ever -- for aggressive maintenance scripts to overload shared >>>>> resources like the databases. But it's back on the table; consider it >>>>> remediated from the temporary guidance "don't do this, it's dangerous" to >>>>> the indefinite guidance "this is powerful, do it cautiously.") >>>>> >>>> >>>> *nods seriously* >>>> >>>> >>>>> Interactive scripts like eval.php can be used with mwscript-k8s in >>>>> --attach mode (which forwards stdin and stdout to your terminal). ( >>>>> https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Input_on_stdin) >>>>> Separately, note there's also mw-debug-repl to get a shell in a mw-debug >>>>> pod, rather than a mw-script one. ( >>>>> https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#MediaWiki_REPL >>>>> ) >>>>> >>>> >>>> *success kid meme* yay! Eval lives. :D >>>> >>>> >>>>> That said, mwscript-k8s can't support {my,}sql.php specifically, >>>>> because we don't have a mysql client in the MW image and shouldn't add >>>>> one. >>>>> For now, you can run `sql` from the deploy hosts instead of the mwmaint >>>>> hosts. The long-term replacement probably won't be a maintenance script at >>>>> all but a dbconfig-aware wrapper around a local mysql client. ( >>>>> https://phabricator.wikimedia.org/T375910) >>>>> >>>> >>>> Awesome, that works fine for my needs. >>>> >>>> Thanks! >>>> >>>> -- brooke >>>> _______________________________________________ >>>> Wikitech-l mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>>> >>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>> >>> _______________________________________________ >>> Wikitech-l mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> >>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >> >> >> >> >> > > -- > Clément 'claime' Goubert (they/them) > Senior SRE > Wikimedia Foundation > -- Clément 'claime' Goubert (they/them) Senior SRE Wikimedia Foundation
_______________________________________________ Wikitech-l mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
