Hello, thanks for the guix bits I am not a sufficiently advanced user of guix repl to do that myself, this is interesting.
> Therefore, the CPU flag might be the same but not the run-time depending > on the machine where it runs. The same binaries in the same pack > produced by Guix might use AVX when available at run-time or do not > otherwise. > > Is it not what you checked when setting on and off using QEMU? To answer your question, I think I misused the term "CPU Flag". When I was saying that I mask the flag, I was talking not of compilation option but really of the flags obtained through lscpu, which corresponds in particular to which acceleration (AVX2 or AVX512) are detected by the binaries run on a given computer. In other words, I change the information on what the binary detect as usable isa at run-time. And from what I understand this is not really emulation because I don't simulate hardware capacity that is not there, I just mask some hardware capacities. I hope I am clear, I am a bit out of my depth here, Timothée ----- Mail original ----- > De: "Simon Tournier" <zimon.touto...@gmail.com> > À: "Timothee Mathieu" <timothee.math...@inria.fr>, "Ludovic Courtès" > <ludovic.cour...@inria.fr> > Cc: "Etienne B. Roesch" <etienne.roe...@gmail.com>, "Andreas Enge" > <andr...@enge.fr>, "Steve George" > <st...@futurile.net>, "Cayetano Santos" <csant...@inventati.org>, "help-guix" > <help-guix@gnu.org>, "Julien Teigny" > <julien.tei...@inria.fr> > Envoyé: Mardi 17 Juin 2025 20:11:08 > Objet: Re: Reproducibility of guix shell container across different host OS > Hi, > > Really interesting use case! > > On Fri, 13 Jun 2025 at 18:38, Timothee Mathieu <timothee.math...@inria.fr> > wrote: > >> I finally concluded that there was a problem with AVX512 somewhere in >> the dependencies graph but I gave up identifying where, as this seems >> very complicated. > > For instance, I had the question: How many packages that python-mujoco > depends on are tunable? > > The answer is none. :-) > > But Guix provides nice tools for manipulating all the graph. For > example, let start an interactive session. > > --8<---------------cut here---------------start------------->8--- > $ cat channels.scm > (list (channel > (name 'guix-science) > (url "https://codeberg.org/guix-science/guix-science.git";) > (branch "master") > (commit > "8bec6b3870dd4589dc4dc7c735f7cc611ac5c947") > (introduction > (make-channel-introduction > "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc" > (openpgp-fingerprint > "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) > (channel > (name 'guix-science-nonfree) > (url "https://codeberg.org/guix-science/guix-science-nonfree/";) > (introduction > (make-channel-introduction > "58661b110325fd5d9b40e6f0177cc486a615817e" > (openpgp-fingerprint > "CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446")))) > (channel > (name 'guix) > (url "https://codeberg.org/guix/guix";) > (branch "master") > (commit > "6f14cf2abc6a3c3bdbf879baa0031fc236cfbe8c") > (introduction > (make-channel-introduction > "9edb3f66fd807b096b48283debdcddccfea34bad" > (openpgp-fingerprint > "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))) > > $ guix time-machine -C channels.scm -- repl -L guix-rl > --8<---------------cut here---------------end--------------->8--- > > From there, it’s possible to traverse all the graph of dependencies and > collect some packages depending on a predicate (PRED) procedure. > > --8<---------------cut here---------------start------------->8--- > scheme@(guix-user)> (use-modules (guix) > (gnu) > (guix sets) > (ice-9 match) > (srfi srfi-1) > ) > > (define (package-inputs* p) > (filter-map (match-lambda > ((_ (? package? x)) x) > (_ #f)) > (package-inputs p))) > > (define (traverse p pred) > (let loop ((lst (list p)) > (result '()) > (visited (setq))) > (match lst > (() > result) > > ((head . tail) > (if (set-contains? visited head) > (loop tail result visited) > (let ((deps (append tail > (package-inputs* head))) > (res (if (pred head) > (cons head result) > result))) > (loop deps res > (set-insert head visited)))))))) > --8<---------------cut here---------------end--------------->8--- > > Then let import the package python-mujoco and check which of its > dependencies (and recursively, dependencies of dependencies, etc.) have > the property tunable?. > > --8<---------------cut here---------------start------------->8--- > scheme@(guix-user)> ,use(guix-rl packages mujoco) > scheme@(guix-user)> (traverse python-mujoco > (lambda (p) > (assq-ref (package-properties p) 'tunable?))) > $1 = () > --8<---------------cut here---------------end--------------->8--- > > And none. Well, I think it’s expected. :-) > > Another example, which packages in the graph of dependencies of > python-mujoco are “replaced”, i.e., grafts. > > --8<---------------cut here---------------start------------->8--- > scheme@(guix-user)> ,pp (map package-full-name (traverse python-mujoco > package-replacement)) > $2 = ("bash-minimal@5.1.16" > "libarchive@3.6.1" > "expat@2.5.0" > "curl@8.6.0") > scheme@(guix-user)> > --8<---------------cut here---------------end--------------->8--- > > In the same idea, it would possible to separate packages depending on > their build system, etc. Or using ’package-mapping‘, rewrite various > fields of some package definition. > > >> I guess all of this should be a cautionary tale that sometimes it may >> be needed to look carefully at the cpu flags in order to get >> reproducibility with guix shell. Ideally I think when we want to have >> something reproducible, we may want to also communicate the cpu flags >> in addition to the manifest and channels file (or at least test that >> changing the flags does not change the results). > > Maybe I miss something, if I understanding correctly [1] > > The solution is to select the most appropriate implementation of > “hot” code at run time. Video players like MPlayer and > number-crunching software like the GNU multiprecision library > have used this “trick” since their inception: using the cpuid > instruction, they can determine at run time which ISA extensions > are available and branch to routines optimized for the available > extensions. Many other applications include similar ad-hoc > mechanism. > > Therefore, the CPU flag might be the same but not the run-time depending > on the machine where it runs. The same binaries in the same pack > produced by Guix might use AVX when available at run-time or do not > otherwise. > > Is it not what you checked when setting on and off using QEMU? > > That said, yes CPU flags might change the result. For a “trivial” > example, the C code, > > #include <stdio.h> > #include <math.h> > > int main(){ > printf("%E\n", j0f(0x1.33d152p+1f)); > } > > compiled with or without the flags ’-lm -fno-builtin’ returns, > > 5.643440E-08 # no options > 5.963430E-08 # -lm -fno-builtin > > because of turning on or off constant folding compiler pass. > > 1: https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/ > >> It would be great if someday a >> feature to mask some CPU flag made its way to guix shell in order to >> improve reproducibility but I guess my case of having a big difference >> due to AVX512 is a limit case that does not happen often (?). > > This would mean that “guix shell” would run on the top of some emulator, > no? > > Yeah I agree, it would nice to be able to launch on the fly an > environment with some specific hardware properties. :-) > > > Cheers, > simon
