Christian, Thanks for this. I've printed it on paper for study and reference. It is slow going for me, but I finally have a feeling that I might have a chance at understanding systemd. The printed version is eleven USletter pages. So far, I've gotten to middle of page three.
Best regards, pec On 20150623_1852+0200, Christian Seiler wrote: > On 06/23/2015 12:59 PM, Erwan David wrote: > > Note that I use policy-rc.d to check whether the encrypted disk is > > mounted for the daemons that need it (it allows not to change the init > > files) > > That works? policy-rc.d should only affect invoke-rc.d, which shouldn't > be relevant at boot, but only in maintainer scripts. (AFAIK at least.) > > > For what I need to know : I have a headless machine with an encrypted disk. > > I cannot ask the password on console, so > > 1) at boot I do not mount the encrypted disk, and start a minimal set > > of daemons, among them the ssh daemon. > > > > 2) I ssh to the machine then mount encrypted disk and start remaining > > daemons. > > > > How can I do this with systemd ? > > This is a great question because it presents a nice little problem that > covers quite a few of topics regarding systemd. I've sat down and > solved your little problem from a systemd perspective, and hopefully my > solution will help you in understanding how systemd works. > > First of all a couple of very basic explanations: what is a unit? A > unit is anything that systemd manages and/or monitors. Relevant for > our case are the types service, target and mount, but systemd supports > more than that (see man systemd.unit and the references therein for > more details). > > A service unit is probably the simplest to understand, it corresponds > to that which was previously provided by /etc/init.d scripts. I'm not > going to go into much detail here about that. > > A mount unit represents a mount in the system. If you manually mount > something via the mount(8) command, systemd will not interfere, but it > will synthesize a dynamic mount unit for it. So for example, if you > have: > mkdir /mnt/a /mnt/b > mount --bind /mnt/a /mnt/b > Then you can also umount /mnt/b via the command: > systemctl stop mnt-b.mount > (umount /mnt/b will continue to work, of course) > > For manual mounts this is not really that relevant (I suspect even most > systemd developers will use plain old umount in that case), but mount > units are the method systemd uses to handle /etc/fstab: for every entry > in /etc/fstab a mount unit is generated, that's how /etc/fstab is > integrated into the boot process. (See below for further details.) > > A target is in some sense the simplest kind of unit: it can only have > dependencies (and a description), but nothing else. You can achieve the > same thing with a dummy /bin/true service unit. Targets are useful for > grouping things together and also for providing synchronization points. > For example, as is documented in man 7 bootup, there's a target called > local-fs.target that has the semantics that every local filesystem > mount in /etc/fstab will be ordered before it, and so that every > service that orders after it can be sure that local filesystems are > already mounted at that point. (Most services are implicitly ordered > after local-fs.target because by default everything is ordered after > basic.target, which itself is ordered after local-fs.target. You can > override these kind of things if you need to, however.) > > > > How does systemd boot up a system? After doing some very basic > initialization (such as setting the hostname from /etc/hostname and > mounting some essential kernel filesystems such as /proc and /sys), it > will try to start a specific unit, either the default (called > 'default.target') or any unit specified on the kernel command line via > the option systemd.unit=XXX. default.target is a symlink to > graphical.target on Debian, but that can be overridden. (If you don't > have a GUI installed, graphical.target is equivalent to > multi-user.target, in analogy to the runlevels 2-5 on Debian with > sysvinit.) graphical.target itself depends on a lot of things, and thus > all services required for system startup are automatically pulled in. > > > > Now how does one solve the problem you have? There are multiple ways to > do so, but for the sake of simplicity I'm going to show just one, the > one I personally prefer (YMMV). > > The basic outline would be this: > > - tell systemd to NOT automatically decrypt the drive at boot and to > NOT automatically mount it > - create a new target unit that will serve as a new boot target, and > it will only contain ssh + syslog as services (beyond the very basic > early-boot things that should always be there) > - create a second target unit that will pull in the encrypted drive > and the mount and then ulitmately start the original > multi-user.target, thus pulling in all other installed daemons on > the system - that second target will be started manually by you > after you log in and decrypt the drive > > > Let's begin. First of all you need to add your drive to /etc/crypttab. > I've done this in a KVM and the entry looks like this: > crypto /dev/vda5 none luks,noauto > (This means that /dev/vda5 is the device with the encrypted LUKS data > on it and /dev/mapper/crypto will be the name of the virtual device > with the plain text data.) > Note that here there is a 'noauto' setting, which means that the drive > should NOT be decrypted automatically at boot (otherwise systemd would > wait for you to enter the password on the console before even mounting > the local filesystems, so it would hang...). > > What happens internally here is that systemd uses a so-called generator > to dynamically transform the contents /etc/crypttab into systemd units. > In this case, it generates a service for each line, in our case it will > generate systemd-cryptsetup@crypto.service. If we had not put noauto > here, the service would be started automatically at boot, asking you > for your password at the console. But with 'noauto' here, the service > will be synthesized by the generator, but not started automatically. > > You can now see if systemd picks up on that crypttab entry be running > systemctl daemon-reload > That does the following: > - re-runs all generators (for reading /etc/crypttab, /etc/fstab, ...) > - re-reads all static and generated unit files on the disk > => so every time you change something, you need to run the above > command for systemd to pick it up > You can see that the unit was detected by systemd via: > systemctl status systemd-cryptsetup@crypto.service > But it should also tell you it's inactive (i.e. hasn't been started so > far). > > Now we add the entry for the filesystem in /etc/fstab: > /dev/mapper/crypto /srv ext4 rw,noauto 0 0 > > Here we also have a noauto setting to make sure it's not automatically > mounted at boot. This is really important here, since this is a local > filesystem and systemd considers any local filesystem that couldn't be > mounted at boot to be a fatal error. So if you leave out the noauto > here, systemd will wait 90s by default for the device to appear at boot > and (since it won't appear) will stop the boot process and start an > emergency shell instead. With a headless system that's a really bad > thing to happen, which is why you should definitely put 'noauto' here. > Then systemd won't automatically try to mount it at boot at all. > > > > > Now that that's done, let's create our target unit for booting the > system with just a minimal set of daemons. We don't want to interfere > with the early-boot process, so we only want to replace the default > graphical.target, but nothing else. > > When editing static units, you should know that systemd knows two > directories where those may be stored: /lib/systemd/system and > /etc/systemd/system. /lib/systemd/system is for things that have been > packaged, whereas /etc/systemd/system is the territory of the > administrator. If files with the same name exist in both directories, > the file in /etc/systemd/system completely overrides the file in > /lib/systemd/system. > > So in this case, we want to edit things in /etc/systemd/system. Let's > create a new file /etc/systemd/system/before-decrypt.target. That file > is supposed to be our new default target, so let's put in the > following: > [Unit] > Description=System before Decryption > Requires=basic.target > Conflicts=rescue.service rescue.target > After=basic.target rescue.service rescue.target > AllowIsolate=yes > > The contents (apart from Description= and Documentation=) is identical > to the default /lib/systemd/system/multi-user.target. (graphical.target > just pulls that in and since we have a headless system, let's skip the > extra indirection.) > > To understand this file, we need to talk systemd dependencies for a > moment. There are a lot of different types of dependencies in systemd, > but in this case you need to understand only a couple (those are also > the most important ones): > > - Requires: if unit A requires the unit B, it means that if A is to > be started, B also has to be started (and if B fails to start, A > will not start either). Also, if B is to be stopped, A will also > be stopped. (Note that this only affects explicit actions, so A > would not be stopped if B segfaults and just dies unexpectedly, > unless you tell systemd to do that with an additional setting.) > > - Wants: if unit A wants the unit B, it means that if A is to be > started, B also has to be started; but A will be started regardless > of whether A fails to start or not (or even exists or not) - and > A will not be stopped if B is stopped. It's a weaker version of > Requires > > - Conflicts: if unit B conflicts with unit B, unit B will be stopped > if unit A is to be started and vice-versa > > - Before/After: declares ordering within a single transaction. Note > that ordering is orthogonal in systemd: Requires/Wants do NOT imply > ordering dependencies, those have to be made explicitly. (Note that > for targets Requires/Wants do sometimes result in implicit ordering, > see below for details; but all other unit types don't have that > logic, so service A requiring service B will NOT lead to an ordering > between them, unless you also specify that explicitly.) > > So in the case of our before-decrypt.target, we have the following > dependencies: > > - Requires=basic.target > This means that the basic system setup should be pulled in. This > is essential; if you leave that out the most basic startup things > will not be started if you try to boot into before-decrypt.target. > > - Conflicts=rescue.service rescue.target > If you were dropped in an emergency shell at boot (or requested it > explicitly over the command line), this conflict will tell systemd > that once you do continue booting from the emergency shell, it > should stop the emergency shell automatically. > > - After=basic.target rescue.service rescue.target > Make sure that before-decrypt.target is started only after the very > basic system initialization is complete. > > But wait a minute: we copied this unit from multi-user.target, where > all the non-GUI services should all be running... Where are they? This > unit file appears to only pull in basic.target and nothing else. > > Well, since you don't want to modify a unit file every time you install > a new daemon on your system, systemd provides ways to add additional > dependencies to units dynamically via symlinks. > > There are two directories for multi-user.target: > /lib/systemd/system/multi-user.target.wants > /etc/systemd/system/multi-user.target.wants > > Each directory contains symlinks to unit files that are automatically > added as Wants= type dependencies to the target (you can also have > .requires). The directory in /lib contains all native systemd services > that shouldn't be disabled by the administrator (and systemctl disable > won't work on them although you could manually do something), while the > latter contains all service that can easily be disabled by the > administrator (those include SSH, cron, MTAs, web servers, ...). > > Typically one wants to still enable all units from the /lib directory > also in our target, so let's do the following: > mkdir /etc/systemd/system/before-decrypt.target.wants > cd /etc/systemd/system/before-decrypt.target.wants > for i in /lib/systemd/system/multi-user.target.wants/* ; do > ln -s /lib/systemd/system/$(basename $i) . > done > Then we additionally want to enable SSH and syslog (but no other > services): > cd /etc/systemd/system/before-decrypt.target.wants > ln -s /etc/systemd/system/syslog.service . > ln -s /lib/systemd/system/ssh.service . > > A couple of notes: > > - we take syslog.service from /etc because I don't know what your > favorite syslog daemon is and syslog.service is always a symlink > to the current one (although you may have to force it manually > because in Jessie there's a Debian packaging bug if you change > syslog daemons that the symlink doesn't get updated) > > - both syslog and ssh are native services, which is why that works > > - you can also enable additional services you might need in the same > way. There's no cron nor atd started here (because those might > require the encrypted partition to be mounted - or not, depending > on your system) and also no MTA > > - note that these symlinks only work if you have native systemd > service files. You can also enable init scripts, but then you have > to explicitly specify Wants=[INITSCRIPTNAME].service, e.g. > Wants=exim4.service - since the service files for init scripts are > dynamically generated and you thus can't easily just create a simple > symlink (in the default case they are activated because systemd's > generator looks for /etc/rcX.d/S**INITSCRIPTNAME symlinks and then > creates Wants= dependencies for runlevelX.target dynamically - and > all runlevel[2-5].target are symlinks to graphical.target) > > Then we should be nearly ready for a test boot to see if our minimal > system boots. To summarize what we have done so far: > > - modify /etc/crypttab and /etc/fstab to add all required entries with > the 'noauto' option > - create /etc/systemd/system/before-decrypt.target (see above for > the contents) > - create directory /etc/systemd/system/before-decrypt.target.wants and > symlink all services in multi-user.target that we need before the > partition is decrypted > > As a final step we can now make our new target the default boot target: > > systemctl set-default before-decrypt.target > > Now let's reboot the system. It will come up again and hopefully the > following will have happened: > > - it didn't hang at boot (otherwise check noauto options) > - SSH was started > - other system daemons that were not included in the list we activated > were NOT started (e.g. no cron/atd/exim4) > > Now for the final step: we want to be able to decrypt the partition and > enter the password. So let's create a file > /etc/systemd/system/decrypt.target that contains the following: > > [Unit] > Description=Decrypted System > Requires=before-decrypt.target > After=before-decrypt.target > Conflicts=systemd-ask-password-console.path > systemd-ask-password-console.service systemd-ask-password-plymouth.path > systemd-ask-password-plymouth.service > Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service > > The interesting settings in that file are: > > - Conflicts= lots of ask-password stuff: > This is to work around a quirk in systemd's handling of asking > passwords for encrypted partitions. See footnote [1] for an > explanation if you are really interested. > > - Requires=...: this is the meat of the matter. Here we make sure we > pull in > - the service that decrypts the partition > - the mount for /srv > - and an additional service (see below) that will cause the rest > of the daemons to start (start-full-system.service) > > This means that when one then starts this target, it will cause the > partition to be decrypted, the filesystem to be mounted and then all > daemons to be started. > > Let's look at start-full-system.service: > [Unit] > Description=Start full system after decryption > After=decrypt.target > > [Service] > Type=oneshot > ExecStartPre=/bin/systemctl is-active --quiet decrypt.target > ExecStart=/bin/systemctl --no-block start multi-user.target > > The unit calls systemctl to start multi-user.target. The reason this > is done is because of ordering: we only want to start daemons once the > filesystem is mounted. But we don't want to add an explicit After= > dependency to each daemon (or list every daemon as a Before= depdency > here), so instead of pulling in multi-user.target directly in > decrypt.target and adding all those dependencies, we simply create a > small service here that is ordered AFTER decrypt.target that causes > systemd to start multi-user.target. > > A word on service ordering: unless you set DefaultDependencies=no, a > target will always be ordered AFTER all of its other dependencies > (Requires=, Wants=, Conflicts=, ...). However, if a service already has > an explicit ordering dependency, this will not occur. So what we have > here is the following ordering: > > decrypt.target has an implicit > After=systemd-cryptsetup@crypto.service srv.mount > BUT because start-full-system.service is ordered relative to > decrypt.mount explicitly already, it doesn't get an implicit ordering. > > So that means we have the following order w.r.t. those 4 units: > > systemd-cryptsetup@crypto.service srv.mount > | | > \-------------------+--------------/ > | > v > decrypt.target > | > v > start-full-system.service > > (In principle, one could also order srv.mount after the cryptsetup > service, but that is unnecessary, since mount units in systemd will > always wait for the corresponding devices to appear, so an explicit > ordering is not required here.) > > If you are wondering what the ExecStartPre= is for: it's to make sure > that decrypt.target was successful before it continues. Otherwise, if > decrypting and/or mounting fails, it would still be executed, so add > an additional check that it only starts if decrypting was > successful.[2] > > As a final thing w.r.t. start-full-system.service, let's talk about > the --no-block here: it causes systemctl to enqueue the start job for > multi-user.target but will return immediately. The reason for that is > that systemd serializes transactions to a certain extent, so that not > using --no-block here would cause the system to deadlock because > systemctl would wait for the multi-user.target transaction to finish > but systemd wouldn't begin the transaction until the transaction > containing start-full-system.service would be finished. (It's generally > a very bad idea to call systemctl without --no-block in service files.) > However, this means that the starting of the daemons will be initiated > by starting decrypt.target, but it will be done in the background. > > > > > > Once this is done, let's do a systemctl daemon-reload to make sure > systemd has picked up the changes and now we can try to see if it > works: > > systemctl start decrypt.target > > You will now be asked for the encrypted partition's password. Once > you've entered that, the filesystem will be mounted, systemctl start > will return you to the command line and in the background all daemons > will be started automatically. > > > > You can reboot the system, log in per SSH again, and use the command > again - and it should work. > > > Now wait a minute: how does this password prompt work exactly? > > - systemd-cryptsetup@XXX.service will tell systemd that it needs a > password to decrypt the drive > > - systemd has to ways of getting to such a passowrd: either via a > so-called 'ask-password agent' (typically used at boot or if the > administrator plugs in a known encrypted drive dynamically) - or, > if the password is asked as part of a transaction that was triggered > via systemct, systemctl will ask the password directly. > > So this means that simply telling systemd to decrypt the drive will > then lead to a direct password prompt at that point. So in the end, one > needs only to enter one single command, then enter the password. > Everything will be decrypted and all services will be started then. > > Neat, huh? > > > > Summary of what we did: > > - adjust /etc/crypttab and /etc/fstab : use noauto! > > - create /etc/systemd/system/before-decrypt.target > > [Unit] > Description=System before Decryption > Requires=basic.target > Conflicts=rescue.service rescue.target > After=basic.target rescue.service rescue.target > AllowIsolate=yes > > - enable required services for before-decrypt.target > > mkdir /etc/systemd/system/before-decrypt.target.wants > cd /etc/systemd/system/before-decrypt.target.wants > for i in /lib/systemd/system/multi-user.target.wants/* ; do > ln -s /lib/systemd/system/$(basename $i) . > done > ln -s /etc/systemd/system/syslog.service . > ln -s /lib/systemd/system/ssh.service . > > - create /etc/systemd/system/decrypt.target > > [Unit] > Description=Decrypted System > Requires=before-decrypt.target > After=before-decrypt.target > Conflicts=systemd-ask-password-console.path > systemd-ask-password-console.service systemd-ask-password-plymouth.path > systemd-ask-password-plymouth.service > Requires=systemd-cryptsetup@crypto.service srv.mount > start-full-system.service > > - create /etc/systemd/system/start-full-system.service > > [Unit] > Description=Start full system after decryption > After=decrypt.target > > [Service] > Type=oneshot > ExecStartPre=/bin/systemctl is-active --quiet decrypt.target > ExecStart=/bin/systemctl --no-block start multi-user.target > > - reboot > > - to decrypt: systemctl start decrypt.target, enter your password at > the prompt > > > > A couple of comments: > > > 1. This can be extended to multiple encrypted drives and also multiple > mounts. If you want to see what unit name systemd wants to use for > a given mount point (so you know what to set as Requires= in > decrypt.target), you can use 'systemctl status /mount/point' to see > how systemd mangles that into a unit name > > 2. If you install additional daemons, Debian will hook them up in > multi-user.target (either directly if they are native or indirectly > if they are sysv scripts), so if you install something that should > already be started before decryption, you need to manually add it. > > On the other hand, since we don't modify multi-user.target itself, > any new daemon will always be hooked up there automatically, so > that if you install something, it will automatically be part of the > start-only-when-decrypted logic.[3] > > 3. If you have systemd installed, it's recommended (but not strictly > required) to have dbus installed. If you have, note that dbus is > the only service (other than systemd's internal ones and some > plymouth stuff that's not relevant for headless systems) that hooks > up in /lib/systemd/system/multi-user.target.wants (and not in > /etc, as all other services do!), and if it's installed it should > probably be activated already for the minimal system, so if you > install dbus only after copying all the symlinks to > before-decrypt.target.wants (see above), then you should copy that > one manually. > > 4. If you storage stack is a bit more complicated (e.g. you use LVM > on top of the encryption), you may or may not need to manually add > some services to activate that once the drive is decrypted. It's > definitely possible, and might even work out of the box (I haven't > tried it), but you may have to play around a bit there. > > 5. If you truly have a headless system, you should test this in a VM > first, especially when it comes to the boot process. > > > > So there it is: how to solve your particular problem with systemd on > Jessie. I hope I could use this use-case to illuminate a couple of > concepts in systemd and give you enough information so that you have a > good starting point into the topic. > > > Regards, > Christian > > > [1] The Conflicts= is required since systemd's own service to notify > root of password prompts via wall(1) has the following command: > /bin/systemctl stop [... all units listed in Conflicts= here ...] > This is fine because it's typically started dynamically when the > administrator plugis in a drive dynamically or so (and is not > active during boot, there the other services are active, which > in turn don't really work after boot, that's why they are stopped). > The problem here is that we start this from within a transaction, > causing systemd to enqueue the stop of the units until after the > transaction for decrypting is done - but then we have a deadlock > again. Using Conflicts= here will simply tell systemd to always > stop those services as part of the transaction to start > decrypt.service, so nothing will deadlock. > > [2] One could also turn it around and NOT make decrypt.target depend > on start-full-system.service, but the other way around (still with > Requires=), then the ExecStartPre would not be necessary, and one > could do systemctl start start-full-system.service... That's > probably a matter of taste... > > [3] There are of course other options of overriding this; for example > one could leave the standard target alone, just override the > services for very few daemons and then hook those up with a new > target. There are advantages and disadvantages with either method. > -- Paul E Condon pecon...@mesanetworks.net -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150624133311.ga3...@big.lan.gnu
