bug#54495: unexpected download after gc
Hello, AFAICT, there is no way to determine which ungrafted package a grafted package comes from without the derivation of the grafted package (where the ungrafted package is referenced). Therefore, I think adding a reference to the ungrafted package in the package itself (your second suggestion) would be the simplest way: https://issues.guix.gnu.org/66824 Presently, it is inconvenient to globally run guix gc at all for me, as many (dependent) packages are deleted and substituted again when rebuilding several profiles built with grafts. Cheers, David
bug#66866: aarch64 system cross compilation + pinebook pro image broken?
Hello,
Mathieu Othacehe writes:
> I could narrow it down somehow.
Starting from alsa-lib, I narrowed it down further. I found that the
problem is actually when an input of the package uses copy-build-system.
In the following example, the cross-compilation (with
--target=x86_64-linux-gnu) for testp2 and testp3 works fine. Their
respective inputs are testinput2 (built with trivial-build-system) and
the hello package (built with gnu-build-system).
For testp1 however, whose input testinput1 is created
with copy-build-system, the same thing as for alsa-lib (which depends on
two packages built with copy-build-system) happens: Bootstrap packages
are starting to be built if grafts are used. The outputs of both
testinput1 and testinput2 do not contain store references, so that is
not the issue. It is also no problem to build the packages testinput1,
testinput2 or hello with grafts.
Is there something special about copy-build-system which is not the case
for trivial-build-system and gnu-build-system? I noticed that in
(guix build-system trival) and (guix build-system gnu), there is a
special case for cross-compilation, while in (guix build-system copy),
the same code is used as for a normal build. I don't see why this could
cause an issue with grafts though.
--8<---cut here---start->8---
(define-module (cc-test)
#:use-module (guix build-system copy)
#:use-module (guix build-system gnu)
#:use-module (guix build-system trivial)
#:use-module (gnu packages base)
#:use-module (guix packages)
#:use-module (guix gexp))
(define-public testinput1
(package
(name "testinput1")
(version "0")
(source
(computed-file
"testp1-src"
#~(begin
(mkdir #$output)
(mkdir (string-append #$output "/dummy")
(build-system copy-build-system)
(arguments (list #:install-plan ''(("dummy" "dummy"
(home-page "")
(synopsis "")
(description "")
(license #f)))
(define-public testinput2
(package
(name "testinput2")
(version "0")
(source #f)
(build-system trivial-build-system)
(arguments
(list
#:builder
#~(begin
(mkdir #$output)
(mkdir (string-append #$output "/dummy")
(home-page "")
(synopsis "")
(description "")
(license #f)))
(define-public testp1
(package
(name "testp1")
(version "0")
(source #f)
(build-system trivial-build-system)
(arguments
(list
#:builder #~(mkdir #$output)))
(inputs (list testinput1))
(home-page "")
(synopsis "")
(description "")
(license #f)))
(define-public testp2
(package
(name "testp2")
(version "0")
(source #f)
(build-system trivial-build-system)
(arguments
(list
#:builder #~(mkdir #$output)))
(inputs (list testinput2))
(home-page "")
(synopsis "")
(description "")
(license #f)))
(define-public testp3
(package
(name "testp3")
(version "0")
(source #f)
(build-system trivial-build-system)
(arguments
(list
#:builder #~(mkdir #$output)))
(inputs (list hello))
(home-page "")
(synopsis "")
(description "")
(license #f)))
--8<---cut here---end--->8---
Best,
David
bug#69922: arcan-sdl does not build
Hello, > While reviewing https://issues.guix.gnu.org/69866, I noticed that > arcan-sdl is failing. I'm recording some notes from looking at the CI > pages here. I meant to reply to this issue, but accidentally opened a new issue with the patch fixing it: https://issues.guix.gnu.org/69923 Should we close this one (69922)? Cheers, David
bug#66866: aarch64 system cross compilation + pinebook pro image broken?
Hi dan, sorry for the late reply. I didn't yet find the reason why different bootstrap packages are built for the cross build when grafts are applied. This seems possibly like another bug to me, as the package inputs which use copy-build-system do not contain store references and it does not happen when they themselves are built with grafts. dan writes: > Can we put everything inside build-inputs? From my understanding, > copy-build-system shouldn't care about cross-compilation at all. I understand it that way too. In guix/build-system.scm, it is mentioned that build/host/target are used in the sense of the GNU toolchain [1]. There, "build" is the system used for building and "host" is the target system for which the package is built. I guess this was confused when copy-build-system was added [2]. Cheers, David [1] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/html_node/Specifying-Target-Triplets.html [2] https://issues.guix.gnu.org/39599
bug#66866: Grafting breaks cross-compilation
Hello, I think I finally understand why the problem only occurs with grafts if a dependency uses copy-build-system. It is actually somewhat complicated. When a package is lowered into a derivation by 'package->derivation' in guix/packages.scm, a list of potentially applicable grafts is created by the 'bag-grafts' procedure. This works by first traversing the bag for native packages (where bag-build-inputs and bag-target-inputs are followed recursively; and also bag-host-inputs if not cross building) and building the native derivations of the grafted versions of the packages which have a replacement (such as glibc at the time of writing). Building the grafted package (in the 'input-graft' procedure) is done again with 'package->derivation', which eventually calls 'graft-derivation' in guix/grafts.scm, which in turn calls 'non-self-references', where the ungrafted package is actually built (ignoring the store monad indirection), not just its derivation. In the case of cross builds, the bag is also traversed for the target dependencies, where the bag-host-inputs are followed recursively. Analogously, this causes the ungrafted packages and their ungrafted replacement to be cross built to compute the grafted derivations. As dan found out, the 'lower' procedure in guix/build-system/copy.scm incorrectly puts standard-packages into the host inputs. They contain the glibc-final package in gnu/packages/commencement.scm (which inherits the replacement of glibc). Because copy-build-system puts them into bag-host-inputs, the package replacement of the glibc-final package is then cross built without grafts when calculating its grafted derivation, which it does not support. Note that the glibc-final package and its package replacement are distinct from the glibc package and its replacement in gnu/packages/base.scm. I confirmed this by running --8<---cut here---start->8--- ,use (guix) (with-store store (run-with-store store (package->cross-derivation (package-replacement (@@ (gnu packages commencement) glibc-final)) "i686-linux-gnu" #:graft? #f))) --8<---cut here---end--->8--- in the REPL, which returns the same incorrect glibc derivation that is attempted to be built as a dependency when running `guix build alsa-lib --target=i686-linux-gnu`. It is actually possible to compute the graft derivations only when needed by a store reference (which does not change the derivation of the grafted package). Then, this problem does not occur even without the fix for the copy-build-system, as the invalid derivations are never actually built. With substitutes or after running GC, this may even prevent building a package with replacement when the requested package output which (transitively) depends on it has no (transitive) store reference to it. I made a patch for that here: https://issues.guix.gnu.org/70895 Regarding the changes to the copy-build-system: Ludovic Courtès writes: > But still, there seem to be some expectation that ‘copy-build-system’ > can support cross-compilation for real, so maybe we should add a > ‘copy-cross-build’ procedure in addition to the patch above. In guix/build-system/copy.scm, it is described as an extension of gnu-build-system and the manual says "It adds much of the gnu-build-system packages to the set of inputs. Because of this, the copy-build-system does not require all the boilerplate code often needed for the trivial-build-system." Therefore, I think it makes sense to add the copy-cross-build procedure so that copy-build-system actually behaves like gnu-build-system. Cheers, David
bug#54495: unexpected download after gc
Hello, David Elsing writes: > Presently, it is inconvenient to globally run guix gc at all for me, as > many (dependent) packages are deleted and substituted again when > rebuilding several profiles built with grafts. In the meantime, I learned about the '--gc-keep-outputs' and '--gc-keep-derivations' options for guix-daemon [1], which together prevent this issue. Of course, this also keeps build dependencies, but that might be desirable anyway. Best, David [1] https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix_002ddaemon.html
bug#75879: [bug#70895] [PATCH] grafts: Only compute necessary graft derivations.
Hello, Ludovic Courtès writes: > Uh, looks like this is a real bug. I’m surprised because we do have > tests for that in ‘tests/gexp.scm’ (and it’s actually used in a few > important places), but maybe they’re not exercising the right thing. Yes indeed, 'with-parameters' is tested for %current-system and %current-target-system, which are evaluated earlier as a special case in the gexp-compiler of , and an additional parameter is only tested by immediately evaluating it. Best, David
bug#75879: with-parameters does not work generally for packages
Hello, I noticed that 'with-parameters' from (guix gexp) does not work with Guile parameters used in package definitions. They are still set in 'lower-object', but not anymore when the monadic procedure returned by 'lower-object' is evaluated. Attached is an example for a package wrapped by 'with-parameters', which results in a file with "D" instead of "C" (it is not "A", because the 'arguments' field of is thunked). Is this intentional? I'm not really sure how (or whether) this should be changed though. Best, David (use-modules (guix build-system trivial) (guix gexp) (guix derivations) (guix packages) (guix store) (gnu packages base)) (define %param (make-parameter "A")) (define testp (package (name "testp") (version "0") (source #f) (build-system trivial-build-system) (arguments (list #:builder #~(let ((port (open-file (string-append #$output) "w"))) (display (string-append #$(%param) "\n") port) (close-port port (home-page #f) (synopsis #f) (description #f) (license #f))) (%param "B") (define obj (with-parameters ((%param "C")) testp)) (%param "D") obj
bug#76110: Broken i686 package on x86_64 since commit 28e4018e59
Hi,
Denis 'GNUtoo' Carikli writes:
> And the process gave the commit 28e4018e59d30efb3d52aa950ce2261f11b69b33
> ("grafts: Allow file-like objects in the ‘replacement’ field of
> .").
>
> However I didn't look into how to repair the behavior above as I'm not
> familiar at all with the code that the 28e4018e59 patch touches.
The problematic change with commit 28e4018e59 is that the 'system'
argument in `input-graft' (and `system' and `target' in
`input-cross-graft') is no longer respected for the replacement, such
that the 'origin' and 'replacement' fields are built for a different
system. Therefore, I would suggest to indeed add another wrapping by
`with-parameters' to `input-graft' and `input-cross-graft' (but keeping
the `with-parameters' wrapping which sets %grafts in
`graft-derivation/shallow', even if it doesn't do anything yet [2]).
Attached is a patch with the change. Unlike other parameters, for which
`with-parameters' currently does not work with packages [2],
`%current-system' and `%current-target-system' are treated specially and
are working correctly.
However, this still does not really solve the issue for your package:
It still (correctly) calls `package->derivation' with the 'system'
argument set to "x86_64-linux", which is then overridden by the #:system
argument in the system package in `bag->derivation'.
The same is not the case however for grafts, and `graft-derivation*' is
still called with "x86_64-linux", which is arguably correct, but
inconsistent with the #:system package argument. IIUC, this leads to
grafts being missed, as some are calculated for "x86_64-linux" (which
are not applicable) and some for "i686-linux".
For reference, is setting #:system in a package even intended? It seems
more coincidental to me that it works, as the #:system argument of a package
overrides the previous #:system argument in `bag->derivation'.
I think it makes more sense to use `with-parameters', which works
correctly in this case (also without the new patch):
--8<---cut here---start->8---
(with-parameters
((%current-system "i686-linux"))
postgresql-14)
--8<---cut here---end--->8---
Cheers,
David
[1] https://issues.guix.gnu.org/70895
[2] https://issues.guix.gnu.org/75879
diff --git a/guix/packages.scm b/guix/packages.scm
index 78726b089ae..43125bac61a 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1824,7 +1824,9 @@ (define (input-graft system)
(return (graft
(origin orig)
(origin-output output)
- (replacement replacement)
+ (replacement
+ (with-parameters ((%current-system system))
+ replacement))
(replacement-output output
package output system)
(return #f
@@ -1846,7 +1848,10 @@ (define (input-cross-graft target system)
(return (graft
(origin orig)
(origin-output output)
- (replacement replacement)
+ (replacement
+ (with-parameters ((%current-system system)
+(%current-target-system target))
+replacement))
(replacement-output output
(return #f
(_
bug#75879: with-parameters does not work generally for packages
Hi Ludo', Ludovic Courtès writes: > Something just came to mind: the object cache. The cache is keyed by > object + system + target + grafts?; if there’s anything that influences > what the object lowers to, changes are the object->derivation mapping is > already cached and that other thing will be ignored. Oh that's true, this is an additional problem. Would it make sense to add the parameters passed to `with-parameters' to the keys used in `mcached'? Then the parameters would have to be passed around quite a bit however... Alternatively, would it be possible to query the fluids with `current-dynamic-state' instead and use that for the keys in `mcached'? I don't see any way to inspect a dynamic state though. > That’s a problem generally speaking with using ‘with-parameters’ with > parameters other than ‘%current-system’, ‘%current-target-system’, and > ‘%graft?’. I think `with-parameters' only works for `%current-system' and `%current-target-system', not for `%graft?': When setting `%graft?' to #f in the second `with-parameters' wrapping in `graft-derivation/shallow', it still evaluates to the same derivation as with `%graft?' set to #t. > I wonder if it’s the only thing at play here though. No, attached is a simplified example without a package using a record (named ) with a (thunked) field. Removing the `mcached' calls in (guix gexp) results in the same derivation. I think this is because `lower-object' returns a monadic procedure, which does not keep the fluids set by `with-parameters', so when the (thunked) field is evaluated, the parameter has changed. Maybe it would be good to pass the parameters from the gexp compiler of to `lower-object' as additional argument and use `with-fluids*' there (just before `lower' is called)? This worked for the record defined in the attachment, but not for a package, I guess because of the expander? I'm not sure how to pass the parameters in that case though. Best, David (use-modules (guix gexp) (guix monads) (guix records) (guix store) (guix utils)) (define-record-type* test make-test test? this-test (field test-field (thunked))) (define* (test->derivation test) (with-monad %store-monad (gexp->derivation "test" #~(let ((port (open-file #$output "w"))) (display (string-append #$(test-field test) "\n") port) (close-port port) (define-gexp-compiler (test-compiler (test ) system target) (test->derivation test)) (define %param (make-parameter "A")) (define testvalue (test (field (%param (%param "B") (define testvalue2 (with-parameters ((%param "C")) testvalue)) (%param "D") testvalue2
bug#76110: Broken i686 package on x86_64 since commit 28e4018e59
Hello, Ludovic Courtès writes: > David, would you be willing/able to send it as a proper patch to > guix-patches, ideally with a test? The patch is here: https://issues.guix.gnu.org/76694 I had to adjust other tests as well, because the replacement can now only be compared by lowering it to a derivation first. Best, David
bug#75879: with-parameters does not work generally for packages
Hi Ludo', Ludovic Courtès writes: > Problem is that the key must remain “small” so that computing its hash > is fast. It cannot grow further than its current size, I’m afraid. What if the hash is calculated in `compile-parameterized' instead (as that is the only supported way to set the parameters) and passed to `lower-object'? > Ideally the key would contain only parameters known to be influential, > but we cannot tell which are influential and which are not among those > in the current dynamic state. Yes that's true, I think the parameters affecting the key should only be the ones passed to `with-parameters', other parameters would then be assumed to be constant. > The solution was to do the same as ‘mparameterize’ to ensure that the > extent during which parameters are set is the correct one given this is > in a monadic context. I think that's a good idea, but I'm not so convinced it makes sense for a general monad [1]. An alternative solution [2] using `with-fluids*' has a different problem however, as it doesn't seem to be compatible with prompts. Thanks, David [1] https://issues.guix.gnu.org/76485#2 [2] https://issues.guix.gnu.org/76485#3
bug#77900: Unprivileged guix-daemon fails to build in Docker/relocatable pack
Hello, Ludovic Courtès writes: > I don’t actually use podman and Docker but I think it would be nice if > the unprivileged guix-daemon would work out of the box in these > environments, particularly in CI environments like GitLab-CI where > passing ‘--security-opt=seccomp=unconfined’ is not an option. Is it not working using `--disable-chroot'? I don't think the isolated build environment is possible when `unshare' is not allowed and the UID is not 0 (except by using something like PRoot), right? > We can ‘unshare’ only once, to lock the mounts inside the build > environment. If that’s the only issue, we could add a command-line > option to disable that or perhaps even detect that we’re in such an > environment and disable it automatically. Ah no, user namespaces can be nested (with a maximum depth of 32), or maybe I'm misunderstanding what you mean? It is just a bit slow to bind mount all directories (and files) in "/" in order to add (or replace) the store, so I added an environment variable inside the chroot in [1]. Cheers, David [1] https://codeberg.org/guix/guix/issues/1054
bug#77900: Unprivileged guix-daemon fails to build in Docker/relocatable pack
Hello, Ludovic Courtès writes: > When running guix-daemon unprivileged in Docker (or, similarly, in a > ‘guix pack -R’ relocatable pack), it fails to spawn the build process: > [...] > The clone(2) man page lists two reasons for getting EPERM with > CLONE_NEWUSER: I'm not sure about `guix pack -R', but I think in the default Docker seccomp profile, the unshare system call [1] requires CAP_SYS_ADMIN, otherwise EPERM is also return. I just tested the Docker seccomp profile with podman, and indeed the unshare command fails because the unshare system call returns EPERM. Maybe you can try with "--security-opt=seccomp=unconfined"? Best, David [1] https://github.com/moby/moby/blob/master/profiles/seccomp/default.json
bug#77900: Unprivileged guix-daemon fails to build in Docker/relocatable pack
Hi, Ludovic Courtès writes: > But it’s unsatisfactory: I would hope the unprivileged daemon would > allow us to address that shortcoming. Yes it does, as long as the needed syscalls are not restricted. I'm not sure when this will change with Docker [1]. >> I don't think the isolated build environment is possible when >> `unshare' is not allowed and the UID is not 0 (except by using >> something like PRoot), right? > > What I meant is that there’s only one ‘unshare’ call, which is necessary > from a security viewpoint but not from a functional viewpoint. Offering > an option to skip it in contexts where the tradeoff is acceptable could > help maybe? Ah sorry, I was conflating `unshare` and `clone`. The default Docker seccomp profile [2] of course also blocks (among other flags) CLONE_NEWUSER (0x1000) for the `clone` syscall without CAP_SYS_ADMIN, using SCMP_CMP_MASKED_EQ. This also leads to EPERM being returned. Best, David [1] https://github.com/moby/moby/issues/42441 [2] https://github.com/moby/moby/blob/master/profiles/seccomp/default.json
bug#77900: Unprivileged guix-daemon fails to build in Docker/relocatable pack
Hi, Ludovic Courtès writes: > So hmm, it looks like in practice we’re left with no choice but to keep > using ‘--disable-chroot’ in Docker? Without unprivileged user namespaces being allowed, the situation hasn't changed I think. > Do you happen to know what people running Docker-in-Docker (or similar) > do? No, but I found this [1] and this [2], so using `--privileged` (or at least allowing unprivileged user namespaces) seems to be necessary. Cheers, David [1] https://docs.docker.com/engine/security/rootless/#rootless-docker-in-docker [2] https://github.com/moby/moby/issues/22139
