On Tue, Apr 15, 2025 at 12:55 PM Zbigniew Jędrzejewski-Szmek
<zbys...@in.waw.pl> wrote:
> [I'm writing this on Colin's suggestion, to introduce more people to
> the discussion and hopefully get some new ideas how to solve the
> problem. Apologies for the length.]
>
> The problem: when packages have files owned by a user or group that
> are part of the package contents ("owned files" for short), and the
> uid/gid is allocated dynamically [1], the uid/gid can change between
> builds of an image from that package ("uid/gid drift") and then in a
> deployment, after an upgrade, the numeric uid/gid that is stored in
> the local system database might be different then the numer used in
> the image, causing files to be owned by the wrong user/group.
>
> This was a known problem for rpm-ostree systems, and was handled
> ad-hoc when problems were reported, but is becoming a bigger problem
> for bootc systems.
>
> There are a few options possible: avoid using any "owned files" in
> packages, use tmpfiles.d to adjust ownership dynamically, switch back
> to soft-static uid/gid assignments, completely change how we do
> deployments, stop upgrading image-based deployments… We need to do
> _something_, but it's not clear at this point what the best course of
> action is.
>
> ** Details of the problem **
>
> I know the summary above is dense. Let's go through an example:
>
> - Package opencryptoki defines group pkcs11. This group uses "dynamic
> allocation", i.e. the specific numeric gid is selected when the
> package is installed, taking the first unused number.
>
> - The package has a file in the %files payload owned by the group:
> %attr(,,pkcs11) /etc/opencryptoki/strength.conf
>
> - When an rpm-ostree or bootc image is built with this package, the
> accounts are created in the local user/group database and some
> numeric uids/gids are selected, for example 999. The filesystem
> stores ownership as numbers, so that gid is used for
> /etc/opencryptoki/strength.conf in the filesystem.
>
> - The user makes a deployment using this image; in the local
> installation the gid is used for /etc/opencryptoki/strength.conf.
>
> - A new version of the rpm-ostree or bootc image is built, but this
> time numeric uids/gids are different, e.g. we get 998 for pkcs11.
>
> - The user upgrades, and the file in the image is owned by a gid that
> maps to a different group in the local database or to no group and
> is listed as unowned.
>
> Note that the package installation order is deterministic and
> stable. So when we rebuild an image with the same package set, we're
> always going to get the same numeric uids/gids. But we can get
> different assignments if the package installation order changes, for
> example because of some dependency change, or more/less packages are
> installed, or when one of the packages defines a new user/group. In
> general, we can expect the uid/gid assignments to change regularly.
>
> The consequences of such wrong ownership can vary: in the most benign
> case this is just a cosmetic issue in 'ls' output. The case that is
> actually reported by users is when the service fails to start because
> it cannot access some file. But then there's also some silent group of
> cases where the wrong ownership creates a security vulnerability or an
> information leak.
>
> This problem exists primarily in "image-based" deployments. In
> traditional "package-based" deployments, uids/gids might be assigned
> differently between systems, but on a given installation, the uid/gid
> assignment remains stable. The problem only appears when the local
> user database is wiped between image builds.
>
> This problem is now primarily visible for rpm-ostree and bootc, but
> would appear also for any image-based system where the image is built
> "remotely" (with a independent user/group database, even if the build
> happens on the same machine), and the creation of local system
> accounts is allowed. One example is ParticleOs [6], where this issues
> is solved by _not_ allowing any "owned files" in the image and using
> tmpfiles.d snippets instead.
>
> "Owned files" also create a problem for image *building*, if this is
> done unprivileged. Mkosi [7] makes 'chown' a noop during unprivileged
> image builds, which means that "owned files" in the image don't work,
> files end up being owned by root.
>
>
> ** UID/GID assignment strategies **
>
> See [1] for a longer discussion, but the tl;dr is that we used to
> assign uids/gids statically, but moved to dynamic assignment about 15
> years ago. The uid/gid range for system accounts is limited, nowadays
> 1-999, in the past 1-499, but we still support upgraded systems with
> the old range. Above that range, we have "user accounts", starting at
> 1000 (or 500 on old systems).
>
> Soft-static uids/gids are taken from the bottom of this range, while
> dynamic uids/gids are taken from the top. Accounts can be created by
> packages, but also by the local admins. See [3] for the list of
> current assignments.
>
> The "soft" part in soft-static is because we never override a uid/gid
> that was assigned locally. Thus if the local admin either used a
> uid/gid for a different purpose, or created the user/group with a
> different uid/gid than our allocation, we honour that.
>
> The soft-static allocations are needed for the case where the
> assignment is shared between systems. In the past this was used with
> some file sharing protocols like nfs/ceph/gluster, where some files
> owned by the system user or group would be shared on the network. But
> nowadays the is mostly used for the initrd: files in /dev or /run are
> owned by various groups to provide unprivelged access, and then after
> the transition to the host system, those file systems are moved over,
> so we need to maitain stability of uids/gids.
>
> In F42+ we have changed the user creation *mechanism* from traditional
> scriptlets which called useradd/groupadd/usermod/groupmod/gpasswd to
> declarative sysusers.d files, which are then executed by calling
> systemd-sysusers [4]. This changes the mechanism, but doesn't change
> the problem with uid/gid drift. systemd-sysusers intentionally
> implements the same soft-static and dynamic allocation strategies
> described in [1]. Arguably, the biggest change is increased
> visibility.
>
> We have 372 unique names in sysusers files in packages in Fedora
> rawhide (for users or groups), and 163 static allocations in setup.
>
>
> ** Potential solutions **
>
> Please note that for the problem to manifest, two conditions are
> necessary: the uid/gid must use dynamic allocation, and the package
> must have "owned files". (Files owned by the user/group but created
> after installation are not a problem.)
>
> Thus the first solution is not to have any owned files. This is the
> recommended approach. It also aligns nicely with the "hermetic /usr"
> initiative, where packages only contain files under /usr. Those files
> must be treated as immutable, and it doesn't make sense for them not
> to be publicly readable, so naturally they are not owned. It also
> aligns nicely with unprivileged builds.
>
> (rpm-ostree/bootc docs recommend systemd's DynamicUser=true. Those
> users are created at runtime with no fixed uid/gid assignement and
> packages which use such users cannot have any owned files. This
> obviously solves the problem too.)
>
> A second "solution" is to keep files "unowned", i.e. owned by root in
> the package payload, but chown them at runtime after installation.
> This can be implemented fairly easily using tmpfiles.d. But it's
> actually quite a bit of work and bother to do correctly. The initial
> ownership must be by root, so that the file does not appear with a
> "wrong" uid/gid after initial installation. Then the ownership needs
> to be adjusted before the service is started, i.e. the service must be
> ordered after a systemd-tmpfiles run. We call systemd-tmpfiles via a
> filetrigger, so normally the case where the package is installed on a
> running system would be handled correctly, i.e. the files would be
> correctly owned after dnf finishes. But this is still fairly fragile.
>
> The third solution is to use soft-static assignment. The most recent
> example of a package where this was added for the purposes of
> rpm-ostree is polkit. In principle we have some space: there aren't
> that many packages with owned files, so we could add static
> assignments for those packages.
>
> Apart from the problem of potentially exhausting the available range,
> soft-static assignments are not great also because they require
> bureaucracy and coordination between different setup and various
> maintainers. This would be a big change to the existing policy and
> would require quite a bit of work to implement and would be of no
> benefit to users of package-based systems. Upgraded systems would not
> get the new assignments, so we would not be able to rely on the new
> static allocations.
>
> Nevertheless, even if we were to switch to soft-static allocations for
> packages, soft-static assignment cannot cover users created by
> non-distro packages and outside of packages. With bootc, it is easy
> for a user to create some user or group in the image (via
> 'RUN groupadd -r foo' or by installing some 3rd-party package).
> So this becomes a bigger problem then previously, see [5].
>
> In fact, we'd probably need to convert _all_ distro packages to
> soft-static allocation, so that those uids/gids are taken from the
> bottom of the range, to keep the top of the range clear for unassigned
> allocations by users using bootc. Even then, users would need to
> carefully maintain the order of operations in their image definitions
> to avoid uid/gid drift.
>
> In the past I suggested some solution where the user/group database
> would be kept betwen rpm-ostree builds. We might have been hard but
> feasible for rpm-ostree, where builds were done in our infra, but it's
> not even feasible for bootc, with many uncoordinated definitions.
>
> We could also consider some bigger changes, like not using numeric
> uids/gids for bootc images, and making the assignment only during
> installation, similarly to how rpm does this…
>
> At this point, I think the approach of combining the first two
> solutions, i.e. minimizing the number of "owned files" and judiciously
> using tmpfiles.d to adjust ownership at runtime is a feasible and
> relatively painless option. To solve the problems for
> rpm-ostree/bootc, we'd need to make this into an official policy and
> adjust some packages.
>
> Comments/ideas?
The problem stems entirely from UIDs and GUDs being numbers
and not strings. I see this as a peculiarity of some of the filesystems,
you target. Say, tar is a filesystem that does not have this problem.
The specific dynamic mapping should thus be stored
within the affected filesystem in question,
and the package manager should implement this mapping as an unpacking quirk.
Simple as that.
On a higher level, I can't help but note that
frivolous ideas like "everyone in the distro
could just switch to use systemd-tmpfiles
for purposes twice removed from the original ones overnight"
or "that concept of file ownership mildly inconveniences us,
and thus should 'simply' be eradicated from the distro",
etc, etc, heard recurrently from the bootc crowd,
are great when you're designing a proper distro from scratch.
You're not.
When a material you're working with is a 20th century distro,
such denial is just an exercise of burying heads in the sand,
leading nowhere. The problem is there.
Unless you personally remove it from every corner of Fedora,
files will be owned.
I can already see elements of bargaining here,
so I wish you to breeze through the depression part as pleasantly as possible
and cut to the acceptance one.
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
