One consequence of this outage is that the server behind the Toolforge Stretch bastion (login-stretch.tools.wmflabs.org) has changed. If you are seeing a scary warning like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8. Please contact your system administrator. Add correct host key in *HOME*/.ssh/known_hosts to get rid of this message. Offending ECDSA key in *HOME*/.ssh/known_hosts:*LINE* ECDSA host key for login-stretch.tools.wmflabs.org has changed and you have requested strict checking. Host key verification failed. then you will need to update your known_hosts file. It probably contains a line like this: login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEMihgdO9CXKJvpoO4LMOt1cU43zIQJiXOm1doVMh0z+uXntQkNDyF\ eHJ9//T983eL8efbCBEgnB9POGfYfoas= You can either change this to login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnJSjCGW7kli+cdgtmndPAl4xLZNc9uqP9KWlsnVDqr8yQ2RkR5ACb\ Xe6XZ+dS09Wc9ulOmGTOwCImMi9Fho78= or remove the line and then look for the following output the next time you SSH into the bastion: The authenticity of host 'login-stretch.tools.wmflabs.org (185.15.56.48)' can't be established. ECDSA key fingerprint is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8. Good luck! Cheers, Lucas Am Mi., 13. Feb. 2019 um 22:53 Uhr schrieb Bryan Davis <bd...@wikimedia.org >: > On Wed, Feb 13, 2019 at 2:45 PM Maximilian Doerr > <maximilian.do...@gmail.com> wrote: > > > > I ask this because of these failures. Where does cyberbot-db-01 live? > > Per <https://tools.wmflabs.org/openstack-browser/project/cyberbot> it > is on cloudvirt1023.eqiad.wmnet > > > The data on there is critical. > > As you probably know, we do not currently have a trusted back up > solution for Cloud VPS projects. Our best recommendation for > 'critical' data is for you to setup some manual or automated backup to > an offsite location (your laptop, a VPS hosted outside Cloud VPS, > etc). Hopefully we will have some news on an actual reliable backup > service in the coming months. We have some hardware to build an > initial system for this, but have not yet had time to design and > implement the backup service itself. > > Bryan > -- > Bryan Davis Wikimedia Foundation <bd...@wikimedia.org> > [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA > irc: bd808 v:415.839.6885 x6855 > > _______________________________________________ > Wikimedia Cloud Services mailing list > Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud -- Lucas Werkmeister Full Stack Developer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
