I just wanted to thank all who responded to my post: the comments were extremely thought-provoking. One thing which came out often and confirmed my own view is that it takes a sysadmin to assess the capabilities of a sysadmin: for management to do so, without the input of the guys at the coal face, is potentially disastrous.
I can't add much to the description of the situation without violating confidentiality constraints. However, I can say a few things about my own position re. this. If it's TL:DR - apologies and no worries. I've worked as a unix and linux sysadmin for over 25 years, and I'm preparing to retire soon (hopefully next year. Quite frankly, given the systemd insanity, I can't wait.). (Before the unix/linux phase, I managed IBM S/370 and S/390 mainframes at various large concerns all over the UK, for 10 years. I still miss mainframes - I suppose in the same way that people miss steam locomotives.) When it was suggested to me that a linux sysadmin might not be able to manage a server running a non-systemd-based linux distro/release, my first reaction was incredulity. It just seemed too preposterous even to contemplate. However, I gradually started to wonder. If it's the case that most linux servers are getting upgraded to systemd-based releases, then obviously the sysadmins managing them will get more exposure to journalctl, sysctl and the other sysadmin tools, and less to the traditional unix/linux tools, and this would obviously be worse in the case of sysadmins who have taken up the profession recently. (But I still can't see that any linux sysadmin worth his/her salt would ever lose touch with the traditional unix/linux tools and techniques altogether.) But in any case, how quickly ***are*** servers being upgraded to systemd-based releases? Throughout my career, it's been my experience that if a server is running a good and stable release of a particular distro (such that the box stays up for years without a problem); and if it's required 24/7 and there are no defined maintenance slots; and if it's relied upon by hundreds of people to do their jobs and earn their living; then the server tends not to be upgraded often. (Releases vary widely - almost like wine vintages: the bad ones get flushed down the drain and forgotten about; the good ones last, and even get better over the years.) Apart from the hassle and interruption to normal usage caused by the upgrade process, there's always the possibility that the new release may introduce new problems and vulnerabilities. In the case of going from a non-systemd to a systemd release, there's (at least) the extra problem of the sysadmin having to learn a complete new set of management tools and procedures, and problem-solving skills, and potentially being less effective and productive during that learning period. Which obviously, given the potential issues from the new release, is just the time when the sysadmin needs to be on top form. In my experience, the only valid motivations for upgrading to a new release have been one or more of: - a new exploitable vulnerability being discovered in the existing release - new hardware being installed, which isn't supported by the existing release - new releases of application software which require functionality not present within the existing release. Giving in to pressure from management to go to the latest shiny new release just for the sake of it is - IMHV - just about the worst case in existence of fixing what ain't broke. (I still remember when I and a number of fellow sysadmins were persuaded to migrate our workstations from (pre-enterprise) Red Hat 6.2 to the brilliant new RH9. After doing so, we discovered that extensive "upgrading and enhancement" had been done on the fonts and font management tools - and as a result, we couldn't read what was on our screens any more. I believe it was the first time that "anti-aliasing" had been rolled out: with it on, the characters were blurs; with it off, they were horribly jagged and multi-coloured. It took several days of fiddling with settings before we could comfortably use GUI-based text tools again, and we never got back to the readability and "look" of the RH6.2 desktop fonts.) I've been hunting for data on internet-facing servers, to try to find out what proportion are now running systemd-based releases. There are plenty of statistics about which distros are running on them, but it seems impossible to go any deeper and find out details about the release levels. If anyone knows a source for this information, I'd be very interested. Apart from my visceral hatred of systemd due to the size and penetration of it, the violation of basic unix principles, and the fact that I can't see why I should have to use it if I don't want to use it, etc. etc. - the main reason why I'm scared of using a systemd-based distro/release on a server is because on several occasions, I've experienced hangs during shutdown and startup when running such a distro/release. The most recent occurrence of this was when I took on the management of a server which was malfunctioning occasionally. I found that at some point - probably accidentally - it had been upgraded to Ubuntu 15, the first fully-systemd release. Also, it was running mysql, and the mysql root directory had been moved to a non-default location. One problem was that mysql would not shut down properly - often, the mysqld process had to be killed manually. One day, the power to the server got interrupted, and it wouldn't come up again. I managed to boot it from a rescue CD, and spent several hours trying to discover what had gone wrong - without success. The systemd "tools" drove me insane - it felt as though they were actively preventing me from accessing the information I needed. Eventually, I tried a hunch: maybe the default credentials required by mysqladmin might not have been copied across when the mysql database directory was moved (one of the nastiest gotchas in running mysql under debian and derivatives). This proved to be the case, and when I'd fixed it, the system came up happily, and a reboot also worked. (As soon as possible, I rolled it back to Ubuntu 14.) The problems were: (a) I'd not been able to find any explicit information about the issue. Obviously, my unfamiliarity with the systemd tools probably didn't help - but it seemed that journalctl was only showing a sanitized and censored version of the available information; (b) systemd would apparently not proceed without confirmation that mysql was up and running. I've had similar mysql-related problems happen on non-systemd systems - but they always came up in spite of the mysql startup problem, and the source of the problem was obvious from the (text-based) logs. In this case, at least I had physical access to the server, and could press the power button and stick a rescue CD into the drive. The thought of the same kind of problem happening on a box in an unattended server farm, or on a VM in a cloud-based hosting environment, with no or limited virtual console access - and of course it would happen in the middle of the night ... Well, it just doesn't bear thinking about! I've said it before and I'll say it again. All this could have been avoided - if systemd had been made optional from day 1. People who liked it could use it; people who didn't like it could use something else. Email traffic to the systemd developers would tend to consist of genuine problem reports and requests for enhancement, rather than hate mail. So there would have been no need for the systemd team to barricade themselves behind a wall of silence and arrogance. And systemd itself would gradually improve. Massive win-win in every conceivable way. The scary thing is that LP and co. must see that as clearly as anyone else. On Fri, 2021-11-19 at 11:29 +0000, Peter Duffy wrote: > I've recently been asked to recommend an upgrade route for a number of > linux servers, and I proposed going to devuan. In response, I've had a > concern raised which took me by surprise. It was suggested that in the > future, it may not be possible to find staff who have the skills to > administer and manage servers running non-systemd or pre-systemd > distros/releases. > > I've tried to give reassurance - but I'm still wondering if this could > be a valid concern. I'd always taken the view that it's primarily the > linux sysadmin community which is trying to stop the onslaught of the > systemd juggernaut - but obviously, the greater the proportion of > servers running systemd-based distros/releases, the less staff get > exposed to non-systemd management techniques and tools. > > I'd be grateful for thoughts and comments. > > _______________________________________________ > Dng mailing list > Dng@lists.dyne.org > https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng _______________________________________________ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
