[gentoo-dev] Per package linker configuration
Hi Everyone, I found that there is some ebuilds that does not respect the "-fuse-ld=..." ldflags(and LD="..."). The only way to change linker for these package is to replace the symlink of "ld" which binutils-config can do that but only for 2 kind of gnu linker (ld.bfd, ld.gold). I can only change to my custom linker by manually symlink it. And above just change the "global" linker, it is hard for user to config liner per package. After dig further, I found that gentoo's libtool/autoconf/automake is already patched (with "libtool-2.4.6-link-fuse-ld.patch") that it allow the "-fuse-ld=..." option pass to compile. But the problem is that these package already autoconfed and automaked by the ebuild maintainer. So there will be a non-patched version of "ltmain.sh" file in the package tarball. The portage only do "./configure" in compile phase (no autoconf, automake) cause the generated Makefile(and its helper script) block the "-fuse-ld=..." options. I create an almost empty stage3 updated(with ~amd64) gentoo box, recompile @world. The below is the list of ebuilds that filter the "-fuse-ld=..." options. (I already excluded ebuilds that mean to filter it, like gcc, glibc...) net-libs/libmnl-1.0.4 app-arch/xz-utils-5.2.3 dev-libs/libltdl-2.4.6 dev-libs/gmp-6.1.2 sys-libs/cracklib-2.9.6-r1 dev-libs/mpfr-3.1.6 dev-libs/mpc-1.0.3 sys-apps/file-5.32 sys-apps/kmod-24 net-firewall/iptables-1.6.1-r2 dev-libs/libpipeline-1.4.2 sys-apps/attr-2.4.47-r2 sys-libs/libseccomp-2.3.2 sys-apps/util-linux-2.31 sys-apps/acl-2.2.52-r1 dev-libs/libpcre-8.41-r1 sys-process/procps-3.3.12-r1 sys-apps/gawk-4.2.0 dev-libs/popt-1.16-r2 net-misc/openssh-7.6_p1 x11-libs/libpciaccess-0.13.5 sys-devel/gettext-0.19.8.1 dev-libs/libgpg-error-1.27-r1 sys-libs/binutils-libs-2.29.1-r1 app-arch/libarchive-3.3.2 sys-apps/man-db-2.7.6.1-r2 sys-libs/pam-1.3.0 Best Regards, Atkins
Re: [gentoo-dev] [v1.0.2] GLEP 74: Full-tree verification using Manifest files
W dniu pon, 30.10.2017 o godzinie 19∶56 +, użytkownik Robin H. Johnson napisał: > On Mon, Oct 30, 2017 at 05:51:36PM +0100, Michał Górny wrote: > ... > > Directory tree coverage > > --- > > This section should maybe cover OPTIONAL in more detail (see more > below). So I've removed OPTIONAL from the new version, so I'm going to skip all the comments regarding it. > > > The Manifest files can also specify ``IGNORE`` entries to skip Manifest > > verification of subdirectories and/or files. The package manager can > > support injecting ignore paths to account for additional files created, > > modified or removed by user's processes that would not be ignored > > by existing rules. Files and directories starting with a dot are always > > implicitly ignored. All files that are not ignored must be covered > > by at least one of the Manifests. > > (English) There are multiple points in this paragraph, and I missed on first > reading. The package manager part is esp. lost. Replaced this with an enumeration. > > > All files that are not ignored must be covered by at least one of the > > > Manifests. Files may be ignored by several ways: > > > - Files and directories starting with a dot are always implicitly > > > ignored. > > > - The Manifest files can specify ``IGNORE`` entries to skip > > > verification of ubdirectories and/or files. > > > - The package manager can support injecting ignore paths to account for > > > additional files created modified or removed by user's processes that > > > would not be ignored by existing rules. > > File verification > > - > > When verifying a file against the Manifest, the following rules are > > used: > > ... > > 3. If the file is covered by an entry of the ``OPTIONAL`` type: > >a. if the file is present, then the verification fails, > >b. otherwise, the verification succeeds. > > Move the OPTIONAL type further up in the verification list maybe? See > the interpretation question below. > > > > Modern Manifest tags > > > > ... > > ``IGNORE `` > > Ignores a subdirectory or file from Manifest checks. If the specified > > path is present, it and its contents are omitted from the Manifest > > verification (always pass). > > Clarification Needed: > Should subdirectories have a trailing slash in the Manifest or not? > This affects matching of the type. > Case 1.1: > Manifest has 'IGNORE foo'; 'foo' is a file; => ignored. > Case 1.2: > Manifest has 'IGNORE foo'; 'foo' is a directory; => ignored. > Case 2.1: > Manifest has 'IGNORE foo/'; 'foo' is a file; => FAIL > Case 2.2: > Manifest has 'IGNORE foo/'; 'foo' is a directory; => ignored. Added a syntax clarification, and noted that we don't support wildcards. > > > > ``OPTIONAL `` > > Specifies a file that does not exist in the distribution but if it > > did, it would be marked as ``MISC``. In the strict mode, the file > > must not exist for the verification to pass. The package manager > > may ignore a stray file matching this entry if operating in non-strict > > mode. > > This has gotten less clear. > Is the following correct interpretation? > if(package manager is strict) then > Treat the OPTIONAL entry as NOT present in the Manifest. > This will cause files to be in the present set but not the covered set. > else > Treat the OPTIONAL entry as 'IGNORE ' > endif > > > ``DIST …`` > > Specifies a distfile entry used to verify files fetched as part > > of ``SRC_URI``. The filename must match the filename used to store > > the fetched file as specified in the PMS [#PMS-FETCH]_. The package > > manager must reject the fetched file if it fails verification. > > ``DIST`` entries apply to all packages below the Manifest file > > specifying them. > > Nit: You have used a unicode ellipsis '…' in some places and plain ASCII > ellipsis '...' in others. Stick to ASCII? Well, I've actually used ASCII only in the code samples because it felt more right but I've switched to Unicode everywhere now, to match dashes. > > > > An example Manifest file (informational) > > > > Can you add an example for OPTIONAL? > > > Tree layout restrictions > > > > The Gentoo repository does not use symbolic links. Some Gentoo > > repositories do, however. To provide a simple solution for dealing with > > symlinks without having to take care to implement special handling for > > them, the common behavior of implicitly resolving them is used. > > Therefore, symbolic links to files are stored as if they were regular > > files, and symbolic links to directories are followed as if they were > > regular directories. > > Clarification: should cross-device symlinks be rejected? (perhaps > implicit, but wanted to check) > If so, need to add to 'Algorithm for full-tree verification' section. Yes, it's implied by the rules in `Directory tree coverage`_. Not sure about adding it t
Re: [gentoo-dev] [v1.0.3] GLEP 74: Full-tree verification using Manifest files
Next version. Now without MISC/OPTIONAL, and with many clarifications. W dniu czw, 26.10.2017 o godzinie 22∶12 +0200, użytkownik Michał Górny napisał: > ReST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst > HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html > impl: https://github.com/mgorny/gemato/ --- GLEP: 74 Title: Full-tree verification using Manifest files Author: Michał Górny , Robin Hugh Johnson , Ulrich Müller Type: Standards Track Status: Draft Version: 1 Created: 2017-10-21 Last-Modified: 2017-10-30 Post-History: 2017-10-26 Content-Type: text/x-rst Requires: 59, 61 Replaces: 44, 58, 60 --- Abstract This GLEP extends the Manifest file format to cover full-tree file integrity and authenticity checks.The format aims to be future-proof, efficient and provide means of backwards compatibility. Motivation == The Manifest files as defined by GLEP 44 [#GLEP44]_ provide the current means of verifying the integrity of distfiles and package files in Gentoo. Combined with OpenPGP signatures, they provide means to ensure the authenticity of the covered files. However, as noted in GLEP 57 [#GLEP57]_ they lack the ability to provide full-tree authenticity verification as they do not cover any files outside the package directory. In particular, they provide multiple ways for a third party to inject malicious code into the ebuild environment. Historically, the topic of providing authenticity coverage for the whole repository has been mentioned multiple times. The most noteworthy effort are GLEPs 58 [#GLEP58]_ and 60 [#GLEP60]_ by Robin H. Johnson from 2008. They were accepted by the Council in 2010 but have never been implemented. When potential implementation work started in 2017, a new discussion about the specification arose. It prompted the creation of a competing GLEP that would provide a redesigned alternative to the old GLEPs. This specification is designed with the following goals in mind: 1. It should provide means to ensure the authenticity of the complete repository, including preventing the injection of additional files. 2. The format should be universal enough to work both for the Gentoo repository and third-party repositories of different characteristics. 3. The Manifest files should be verifiable stand-alone, that is without knowing any details about the underlying repository format. Specification = Manifest file format This specification reuses and extends the Manifest file format defined in GLEP 44 [#GLEP44]_. For the purpose of it, the *file type* field is repurposed as a generic *tag* that could also indicate additional (non-checksum) metadata. Appropriately, those tags can be followed by other space-separated values. Unless specified otherwise, the paths used in the Manifest files are relative to the directory containing the Manifest file. The paths must not reference the parent directory (``..``). Manifest file locations and nesting --- The ``Manifest`` file located in the root directory of the repository is called top-level Manifest, and it is used to perform the full-tree verification. In order to verify the authenticity, it must be signed using OpenPGP, using the armored cleartext format. The top-level Manifest may reference sub-Manifests contained in subdirectories of the repository. The sub-Manifests are traditionally named ``Manifest``; however, the implementation must support arbitrary names, including the possibility of multiple (split) Manifests for a single directory. The sub-Manifest can only cover the files inside the directory tree where it resides. The sub-Manifest can also be signed using OpenPGP armored cleartext format. However, the signature verification can be omitted if it is covered by a signed top-level Manifest. Directory tree coverage --- The specification provides three ways of skipping Manifest verification of specific files and directories (recursively): 1. explicit ``IGNORE`` entries in Manifest files, 2. injected ignore paths via package manager configuration, 3. using names starting with a dot (``.``) which are always skipped. All files that are not ignored must be covered by at least one of the Manifests. A single file may be matched by multiple identical or equivalent Manifest entries, if and only if the entries have the same semantics, specify the same size and the checksums common to both entries match. It is an error for a single file to be matched by multiple entries of different semantics, file size or checksum values. It is an error to specify another entry for a file matching ``IGNORE``, or one of its subdirectories. The file entries (except for ``IGNORE``) can be specified for regular files only. Symbolic links are followed when opening files and traversing directories. It is an error to specify an entry for a different file type. If the tree contain files of other types that are not otherw
Re: [gentoo-dev] [v1.0.3] GLEP 74: Full-tree verification using Manifest files
On Thu, Nov 02, 2017 at 08:11:59PM +0100, Michał Górny wrote: > Next version. Now without MISC/OPTIONAL, and with many clarifications. Huge improvements in this version, I found it much easier to understand. Nits: - please stick to ASCII ellipsis. The unicode ellipsis is unreadable in some monospace fonts. Further items inline: > Directory tree coverage > --- ... > The file entries (except for ``IGNORE``) can be specified for regular > files only. Symbolic links are followed when opening files > and traversing directories. It is an error to specify an entry for > a different file type. If the tree contain files of other types > that are not otherwise ignored, they need to be covered by an explicit > ``IGNORE``. > > All the local (non-``DIST``) files covered by a Manifest tree must > reside on the same filesystem. It is an error to specify entries > applying to files on another filesystem. If subdirectories > that are not otherwise ignored reside on a different filesystem, they > must be explicitly excluded via ``IGNORE``. I would prefer this to say: 'If files that are not otherwise ignored reside on a different filesystem', as expanded from sub-directories. This implicitly forbids following a symlink that crosses a filesystem boundary, and then matches the similar part of 'Tree layout restrictions'. > Rationale > = ... > Tree layout restrictions > > > The algorithm is meant to work primarily with ebuild repositories which > normally contain only files and directories. Directories provide > no useful metadata for verification, and specifying special entries > for additional file types is purposeless. Therefore, the specification > is restricted to dealing with regular files. > > The Gentoo repository does not use symbolic links. Some Gentoo > repositories do, however. To provide a simple solution for dealing with > symlinks without having to take care to implement special handling for > them, the common behavior of implicitly resolving them is used. > Therefore, symbolic links to files are stored as if they were regular > files, and symbolic links to directories are followed as if they were > regular directories. > > Dotfiles are implicitly ignored as that is a common notion used > in software written for POSIX systems. All other common filenames > require explicit ``IGNORE`` lines. 'common' in the second sentence seems odd. What about uncommon filenames? Maybe just s/other common filenames/other filenames/. > An ability to inject additional ignore entries is provided to account > for site configuration affecting the repository tree — placing > additional files in it, skipping some of the categories from syncing. Mention that the package manager may provide wildcards or regex in the additional entries. Eg: 'IGNORE **/metadata.xml' > Non-strict Manifest verification > ... > The cases for stripping unnecessary files mostly focused around space > savings. For this purpose, stripping ``metadata.xml`` and similar files > has little value. It is much more common for users to strip whole > categories which can not be handled via the ``MISC`` type, and needs > a dedicated package manager mechanism. The same mechanism can also > handle files that used the ``MISC`` type. Exclusion by package does happen as well. A list of categories or packages can be used for both the rsync exclusion and the IGNORE. > Splitting distfile checksums from file checksums > > > Another problem with the current Manifest format is that the checksums > for fetched files are combined with checksums for local files > in a single file inside the package directory. It has been specifically > pointed out that: > > - since distfiles are sometimes reused across different packages, > the repeating checksums are redundant, Comment: 8.4% of all DIST entries are duplicate, representing a 2MiB saving in tree size (25MiB of DIST entries altogether). > - mirror admins were interested in the possibility of verifying all > the distfiles with a single tool. > > This specification does not provide a clean solution to this problem. > It technically permits moving ``DIST`` entries to higher-level Manifests > but the usefulness of such a solution is doubtful. This solution would require the packager manager to consider higher-level Manifests or all Manifests in the tree when searching for the DIST entry. The most useful implementation of this would be for the git->rsync process to move all DIST entries elsewhere (metadata/ maybe). Either way, this would have many downsides, and make manual work on the Manifest DIST entries painful. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature
