On Wed, 2010-01-13 at 09:45 -0600, Bryce T. Pier wrote: > For years my employer has only patched *nix systems on an annual basis. > We've now been directed to apply security patches quarterly. Due to the > infrequency of patching in the past, there has developed a fairly high > level of paranoia around patching "breaking" things, particularly > related to servers not coming back from the post-patch reboot. To > mitigate these fears I've been asked to document procedures for > backing-out the applied patches and/or recovering the server in the > event of one not coming back up. > > Given that tools like RHN Satellite or Novell Zenworks don't have the > ability to do extensive pre-patch preparations like breaking hardware > root mirrors or running filesystem dumps, I have the impression that at > least in enterprise Linuxes there aren't frequently issues caused by > normal, regular patching activities. > > So I'm curious what other people are doing on the Linux platform. > > Do you use root disk mirrors and break the mirror prior to patching? > Do you utilize filesystem dumps (dumpe2fs, etc) or rely on enterprise > backups of the OS filesystems? > Do you use rpm rollbacks? > Rebuild / re-image the server if there are problems? > > Additionally, have you experienced many instances of patching tanking an > enterprise Linux server in the last couple of years? > > Thanks much!
At work, the Linux systems are mostly my responsibility. WE use the Scientific Linux distribution (RHEL 5.X). We go have some highly critical Linux systems, but they are mostly single function systems (DNS, LDAP, list server, etc). So they are all fairly easy to kept patched up. Which I do quite often. Saving it all for quarter, annualy, is just too much work at one time. Patching more often actually takes less amount of time in the long run in my experience. So have enterprise backup systems which backs up everything, so we have that to fall back on. Every files saved for at least 1 year, and 2 versions of those files. But with the single function nature of the systems, I can back up updates by reinstalling the previous versions of RPMS. And we have gotten into some virtualization. My last hardware refresh got me system with a lot of excess capacity (end of year deals). So I'm running XEN on a number of systems. And we've moved a number of single server over to XEN instances. With XEN and LVM2 r/w snapshots, I can create a virtual clone of a system, and play with it to my hearts content. Upgrade it and testing it extensively w/o having to affect the running production system. In the worst case scenario, we fall back to doing a basic OS install, install the TSM backup client, and restore the system from backups. I have had to do this once. We had a list server running on an out of maintenance server (didn't know it at the time). Server was dead after a power event in our data center. I was able to create a virtual server on one of my beefy XEN servers. Restore the data from backup and get back into production in a few hours. It worked out quite nice. I also for an OS upgrade for the server out of the deal. :) -- Stephen Johnson <sjohn...@monsters.org> _______________________________________________ Discuss mailing list Discuss@lopsa.org http://lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
