Hi folks,
[I'm writing this on Colin's suggestion, to introduce more people to
the discussion and hopefully get some new ideas how to solve the
problem. Apologies for the length.]
The problem: when packages have files owned by a user or group that
are part of the package contents ("owned files" for short), and the
uid/gid is allocated dynamically [1], the uid/gid can change between
builds of an image from that package ("uid/gid drift") and then in a
deployment, after an upgrade, the numeric uid/gid that is stored in
the local system database might be different then the numer used in
the image, causing files to be owned by the wrong user/group.
This was a known problem for rpm-ostree systems, and was handled
ad-hoc when problems were reported, but is becoming a bigger problem
for bootc systems.
There are a few options possible: avoid using any "owned files" in
packages, use tmpfiles.d to adjust ownership dynamically, switch back
to soft-static uid/gid assignments, completely change how we do
deployments, stop upgrading image-based deployments… We need to do
_something_, but it's not clear at this point what the best course of
action is.
** Details of the problem **
I know the summary above is dense. Let's go through an example:
- Package opencryptoki defines group pkcs11. This group uses "dynamic
allocation", i.e. the specific numeric gid is selected when the
package is installed, taking the first unused number.
- The package has a file in the %files payload owned by the group:
%attr(,,pkcs11) /etc/opencryptoki/strength.conf
- When an rpm-ostree or bootc image is built with this package, the
accounts are created in the local user/group database and some
numeric uids/gids are selected, for example 999. The filesystem
stores ownership as numbers, so that gid is used for
/etc/opencryptoki/strength.conf in the filesystem.
- The user makes a deployment using this image; in the local
installation the gid is used for /etc/opencryptoki/strength.conf.
- A new version of the rpm-ostree or bootc image is built, but this
time numeric uids/gids are different, e.g. we get 998 for pkcs11.
- The user upgrades, and the file in the image is owned by a gid that
maps to a different group in the local database or to no group and
is listed as unowned.
Note that the package installation order is deterministic and
stable. So when we rebuild an image with the same package set, we're
always going to get the same numeric uids/gids. But we can get
different assignments if the package installation order changes, for
example because of some dependency change, or more/less packages are
installed, or when one of the packages defines a new user/group. In
general, we can expect the uid/gid assignments to change regularly.
The consequences of such wrong ownership can vary: in the most benign
case this is just a cosmetic issue in 'ls' output. The case that is
actually reported by users is when the service fails to start because
it cannot access some file. But then there's also some silent group of
cases where the wrong ownership creates a security vulnerability or an
information leak.
This problem exists primarily in "image-based" deployments. In
traditional "package-based" deployments, uids/gids might be assigned
differently between systems, but on a given installation, the uid/gid
assignment remains stable. The problem only appears when the local
user database is wiped between image builds.
This problem is now primarily visible for rpm-ostree and bootc, but
would appear also for any image-based system where the image is built
"remotely" (with a independent user/group database, even if the build
happens on the same machine), and the creation of local system
accounts is allowed. One example is ParticleOs [6], where this issues
is solved by _not_ allowing any "owned files" in the image and using
tmpfiles.d snippets instead.
"Owned files" also create a problem for image *building*, if this is
done unprivileged. Mkosi [7] makes 'chown' a noop during unprivileged
image builds, which means that "owned files" in the image don't work,
files end up being owned by root.
** UID/GID assignment strategies **
See [1] for a longer discussion, but the tl;dr is that we used to
assign uids/gids statically, but moved to dynamic assignment about 15
years ago. The uid/gid range for system accounts is limited, nowadays
1-999, in the past 1-499, but we still support upgraded systems with
the old range. Above that range, we have "user accounts", starting at
1000 (or 500 on old systems).
Soft-static uids/gids are taken from the bottom of this range, while
dynamic uids/gids are taken from the top. Accounts can be created by
packages, but also by the local admins. See [3] for the list of
current assignments.
The "soft" part in soft-static is because we never override a uid/gid
that was assigned locally. Thus if the local admin either used a
uid/gid for a different purpose, or created the user/group with a
different uid/gid than our allocation, we honour that.
The soft-static allocations are needed for the case where the
assignment is shared between systems. In the past this was used with
some file sharing protocols like nfs/ceph/gluster, where some files
owned by the system user or group would be shared on the network. But
nowadays the is mostly used for the initrd: files in /dev or /run are
owned by various groups to provide unprivelged access, and then after
the transition to the host system, those file systems are moved over,
so we need to maitain stability of uids/gids.
In F42+ we have changed the user creation *mechanism* from traditional
scriptlets which called useradd/groupadd/usermod/groupmod/gpasswd to
declarative sysusers.d files, which are then executed by calling
systemd-sysusers [4]. This changes the mechanism, but doesn't change
the problem with uid/gid drift. systemd-sysusers intentionally
implements the same soft-static and dynamic allocation strategies
described in [1]. Arguably, the biggest change is increased
visibility.
We have 372 unique names in sysusers files in packages in Fedora
rawhide (for users or groups), and 163 static allocations in setup.
** Potential solutions **
Please note that for the problem to manifest, two conditions are
necessary: the uid/gid must use dynamic allocation, and the package
must have "owned files". (Files owned by the user/group but created
after installation are not a problem.)
Thus the first solution is not to have any owned files. This is the
recommended approach. It also aligns nicely with the "hermetic /usr"
initiative, where packages only contain files under /usr. Those files
must be treated as immutable, and it doesn't make sense for them not
to be publicly readable, so naturally they are not owned. It also
aligns nicely with unprivileged builds.
(rpm-ostree/bootc docs recommend systemd's DynamicUser=true. Those
users are created at runtime with no fixed uid/gid assignement and
packages which use such users cannot have any owned files. This
obviously solves the problem too.)
A second "solution" is to keep files "unowned", i.e. owned by root in
the package payload, but chown them at runtime after installation.
This can be implemented fairly easily using tmpfiles.d. But it's
actually quite a bit of work and bother to do correctly. The initial
ownership must be by root, so that the file does not appear with a
"wrong" uid/gid after initial installation. Then the ownership needs
to be adjusted before the service is started, i.e. the service must be
ordered after a systemd-tmpfiles run. We call systemd-tmpfiles via a
filetrigger, so normally the case where the package is installed on a
running system would be handled correctly, i.e. the files would be
correctly owned after dnf finishes. But this is still fairly fragile.
The third solution is to use soft-static assignment. The most recent
example of a package where this was added for the purposes of
rpm-ostree is polkit. In principle we have some space: there aren't
that many packages with owned files, so we could add static
assignments for those packages.
Apart from the problem of potentially exhausting the available range,
soft-static assignments are not great also because they require
bureaucracy and coordination between different setup and various
maintainers. This would be a big change to the existing policy and
would require quite a bit of work to implement and would be of no
benefit to users of package-based systems. Upgraded systems would not
get the new assignments, so we would not be able to rely on the new
static allocations.
Nevertheless, even if we were to switch to soft-static allocations for
packages, soft-static assignment cannot cover users created by
non-distro packages and outside of packages. With bootc, it is easy
for a user to create some user or group in the image (via
'RUN groupadd -r foo' or by installing some 3rd-party package).
So this becomes a bigger problem then previously, see [5].
In fact, we'd probably need to convert _all_ distro packages to
soft-static allocation, so that those uids/gids are taken from the
bottom of the range, to keep the top of the range clear for unassigned
allocations by users using bootc. Even then, users would need to
carefully maintain the order of operations in their image definitions
to avoid uid/gid drift.
In the past I suggested some solution where the user/group database
would be kept betwen rpm-ostree builds. We might have been hard but
feasible for rpm-ostree, where builds were done in our infra, but it's
not even feasible for bootc, with many uncoordinated definitions.
We could also consider some bigger changes, like not using numeric
uids/gids for bootc images, and making the assignment only during
installation, similarly to how rpm does this…
At this point, I think the approach of combining the first two
solutions, i.e. minimizing the number of "owned files" and judiciously
using tmpfiles.d to adjust ownership at runtime is a feasible and
relatively painless option. To solve the problems for
rpm-ostree/bootc, we'd need to make this into an official policy and
adjust some packages.
Comments/ideas?
[1]
https://docs.fedoraproject.org/en-US/packaging-guidelines/UsersAndGroups/#_dynamic_allocation
[2] https://github.com/uapi-group/specifications/issues/76
[3] https://src.fedoraproject.org/rpms/setup/blob/rawhide/f/uidgid
[4] https://fedoraproject.org/wiki/Changes/RPMSuportForSystemdSysusers
[5] https://issues.redhat.com/browse/RHEL-77146
[6] https://github.com/systemd/particleos
[7] https://github.com/systemd/mkosi
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
