At a previous job we managed to push out passwd file to several hundred servers without a root account in it. (we'd forgotten to make root a protected account that could never expire in the generating script we used with cfengine) That was fun. All sorts of stuff broke in some very interesting ways. That lead to a fun day of running around servers with recovery disks and replacing the passwd and shadow files.
David Lang <da...@lang.hm> wrote: >to err is human, to really foul things up requires a computer > >...and when you automate changes to computers.... > >I've done similar things, not reformatting everything, but I managed to use >an automation tool to break all 250 firewalls in at $prior_job in a way that >disabled the automation at the same time, requiring booting from recovery >media >and manual changes to each box to recover. To complicate things, the firewalls >mostly continued to work, so we had to juggle the fixes to avoid breaking >things >even worse. > >The good news was that the automation was good enough that I was able to give >a >couple people instructions on how to recover and we got everything fixed in a >few hours, but it was an interesting afternoon. > >David Lang > >On Sun, 18 May 2014, Nick Webb wrote: > >> On Sun, May 18, 2014 at 9:38 PM, David Lang <da...@lang.hm> wrote: >> >>> wayback to the rescue >>> >>> http://web.archive.org/web/20140516225155/http://it. >>> emory.edu/windows7-incident/ >>> >>> >> I hang my head in shame for not checking there! >> >> Wow this is/was a nightmare. For those of us working on automation >> initiatives, this is one downside to be careful of... when it's so easy to >> make a mass change we must take extra care... >> >_______________________________________________ >Discuss mailing list >Discuss@lists.lopsa.org >https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss >This list provided by the League of Professional System Administrators > http://lopsa.org/ _______________________________________________ Discuss mailing list Discuss@lists.lopsa.org https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
