Re: Better support remote deployment
Hello, Felix Lechner writes: > Hi, > > On Tue, Nov 07 2023, Maxim Cournoyer wrote: > >> Then your suggestion to extend 'guix package' to be able to >> install a profile from a store profile sounds useful, or perhaps a new >> 'guix deploy-profile' (or a better name / sub-command?) that would >> automate the copy and installation in one step. > > We have a lot of sub-commands already. Could 'guix deploy' instead be > fed something other than a 'machine' record? Perhaps too much of an overload of that command; not sure. > How about a 'remote-user-profile'? 'guix package' is already the command we use to create profiles; maybe it could accept a '--remote' argument to operate on a remote machine? Or is this not what 'GUIX_DAEMON_SOCKET=ssh://your-target guix package -m manifest.scm' can already accomplish? :-) -- Thanks, Maxim
Re: Turning off tests leads to a different store item
On 2023-11-08, Felix Lechner via wrote: > On Wed, Nov 08 2023, Maxim Cournoyer wrote: >> A source tree doesn't produce a derivation. A derivation is the >> complete build recipe that captures the source and the package >> definition, that when built by the daemon produces a store item. > > Okay, thanks! Now I'm going to get it right: > > The store item that is produced should not change whether build-time > tests run or not. > > It does not make sense (and wastes resources) to rebuild a consuming > package because build-time tests were enabled or disabled in an input. > > The historical version of openssl gave rise to this thread. It did not > build anymore because the tests no longer worked with the certificates > shipped in that release (a common problem in TLS libraries). Rebuilding > openssl without running the tests rendered the rebuild useless because > it produced a different store item. That should not happen. > > Does that make more sense? I do not really think people are misunderstanding you, more that your *should* does not align with reality in a way that guix can depend on; build-time tests *do* affect the build result in some cases. You can write tests that can be run outside the build environment, like Debian's autopkgtest, but those may need to actually be different tests and most upstreams do not provide them yet... The only way to ensure they do not change the build results is even more resource-intensive; systematically run the build without tests and then run the build with tests and compare the store items... assuming the build is reproducible in the first place, which is not yet 100% reliable either, unfortunately. This does seem well outside the context of help-guix (drop from CC in replies?) at this point, and should perhaps be moved to guix-devel(added to CC), as it may require significant changes to guix; it is not a simple matter of helping someone figure out how to do something with guix. live well, vagrant signature.asc Description: PGP signature
Re: Turning off tests leads to a different store item
On Wed, Nov 08, 2023 at 10:18:40AM -0800, Vagrant Cascadian wrote: > On 2023-11-08, Felix Lechner via wrote: > > On Wed, Nov 08 2023, Maxim Cournoyer wrote: > >> A source tree doesn't produce a derivation. A derivation is the > >> complete build recipe that captures the source and the package > >> definition, that when built by the daemon produces a store item. > > > > Okay, thanks! Now I'm going to get it right: > > > > The store item that is produced should not change whether build-time > > tests run or not. > > > > It does not make sense (and wastes resources) to rebuild a consuming > > package because build-time tests were enabled or disabled in an input. > > I do not really think people are misunderstanding you, more that your > *should* does not align with reality in a way that guix can depend on; > build-time tests *do* affect the build result in some cases. > > [...] > > The only way to ensure they do not change the build results is even more > resource-intensive; systematically run the build without tests and then > run the build with tests and compare the store items... assuming the > build is reproducible in the first place, which is not yet 100% reliable > either, unfortunately. There is another way: simply preventing the tests from changing the resulting store item. For example, the package could first be built without tests and then that build tree could be copied to the build tree of the build with tests enabled. The result of that build could then just be copied from the testless build, ignoring any changes the test suite has made to the build tree. I'm not confident enough in my understanding of how Guix builds things to say for sure that this specific method would work, but I am quite sure that the general idea is implementable. signature.asc Description: PGP signature
Re: Turning off tests leads to a different store item
Hi, On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori wrote: > There is another way: simply preventing the tests from changing the > resulting store item. For example, the package could first be built > without tests and then that build tree could be copied to the build tree > of the build with tests enabled. Somehow, the store would need to keep all the build intermediary artifacts produced, right? For instance, consider the extreme case where the build phase produces .o artifact files and the tests for whatever reasons relies on these temporary artifacts. Well, we had a quick chat with Josselin and Andreas about separating the tests at https://hpc.guix.info/events/2023/workshop/program/ And my understanding of the rough conclusion we had: it is not easy and the evil are in all the details. For example autotools: somehow "make check" is connected in one way or the other to "make" and/or "make install". Somehow, the complete build tree (with intermediary artifacts as .o) should keep in the store. >From a pragmatical point of view, there is packages where the tests cannot be totally separated from from the temporary build, therefore the question seems: how do these cases compare to the other regular cases? What is the ratio? Is the rule about many corner cases without a clear "regular"? Or are they just few corner cases? Cheers, simon
Stand Guix au Capitole du Libre à Toulouse, nov. 18-19
Salut, La date du Capitole du Libre approche... Chouette ! L'organisation Capitole du Libre demande le nombre de personnes qui aideront pour le stand (une histoire de sécurité et de badge). Il serait aussi judicieux que l’on puisse avoir une idée entre nous pour faciliter la journée ou les journées. Qu’est-ce que cela signifie être au stand ? Parler de Guix et de logiciel libre. Rien de formel et pas besoin d’être un expert. :-) Si vous lisez cet email, cela signifie que vous êtes suffisamment intéressé par Guix pour en discuter avec des gens qui ne connaissent pas du tout. Par exemple, les créneaux qui me conviendraient – rien de fixe non plus ;-) – présence samedi matin pour l’installation jusqu’à 13h, vadrouille jusqu’à 15h, puis 15h-19h. J’aimerais arriver le dimanche vers 11h, puis jusqu’au démontage. Bref, voilà l’idée quoi. :-) Vous pouvez m’envoyer un email en privé si vous préférez. Et cela ne vous engage pas de manière ferme non plus. À très vite, simon
Recommended way of packaging web themes
Hi, With "web theme" I mean a directory containing CSS, JavaScript, images and other files that define a particular look and feel for a website, web application or other kinds of products that use web technologies. For example, I'm looking into packaging the current Guix's website theme and a new theme that I've been planning to develop (based on the former). If these were packaged, I think it would be easier for other Guix websites and services (e.g. CI, QA, DATA) to use them and hopefully achieve a more uniform look and feel for the network of Guix web fronts. I've experimented a bit and packaged both themes: https://codeberg.org/luis-felipe/guix-channel-x/src/branch/trunk/luflac-x/packages/web-themes.scm Once installed, I can access both themes from a custom path indicated by a non-standard environment variable (WEB_THEMES_PATH, currently set by the defined packages): #+begin_example ❯ guix shell -L luflac-x guile guix-website-theme-initial guix-website-theme-aureus ❯ guile ... > (getenv "WEB_THEMES_PATH") $1 = "/gnu/store/gc0szk5h3s45d605ikzyxxxc4z2b9l3l-profile/www/themes" > (define guix-icon-path (string-append $1 "/aureus/base/img/icon.svg")) > (file-exists? guix-icon-path) $2 = #t #+end_example However, while this seems to work alright, I'm not confident it is the right thing to do... How would you recommend packaging these kinds of themes? Are there any standard paths and environment variables for web themes? Or should one use Freedesktop.org's XDG_DATA_DIRS and their specified locations instead? Thanks in advance, -- Luis Felipe López Acevedo https://luis-felipe.gitlab.io/ OpenPGP_0x0AB0D067012F08C3.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: Recommended way of packaging web themes
Hi Felipe, On Wed, Nov 08 2023, Luis Felipe wrote: > I'm looking into packaging the current Guix's website theme > [...] I think it would be easier for other > Guix websites and services (e.g. CI, QA, DATA) to use them and hopefully > achieve a more uniform look and feel for the network of Guix web fronts. Could you simply publish the resources online (except Javascript, of course) so any site can use them? It would be similar to Google Fonts. [1] Then I would package the resources being served in such a way that anyone can easily set up a second server. Kind regards Felix [1] https://fonts.google.com/
Re: Turning off tests leads to a different store item
Hi, Simon Tournier writes: > Hi, > > On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori wrote: > >> There is another way: simply preventing the tests from changing the >> resulting store item. For example, the package could first be built >> without tests and then that build tree could be copied to the build tree >> of the build with tests enabled. > > Somehow, the store would need to keep all the build intermediary > artifacts produced, right? For instance, consider the extreme case > where the build phase produces .o artifact files and the tests for > whatever reasons relies on these temporary artifacts. > > Well, we had a quick chat with Josselin and Andreas about separating > the tests at https://hpc.guix.info/events/2023/workshop/program/ > And my understanding of the rough conclusion we had: it is not easy > and the evil are in all the details. For example autotools: somehow > "make check" is connected in one way or the other to "make" and/or > "make install". Somehow, the complete build tree (with intermediary > artifacts as .o) should keep in the store. > > From a pragmatical point of view, there is packages where the tests > cannot be totally separated from from the temporary build, therefore > the question seems: how do these cases compare to the other regular > cases? What is the ratio? Is the rule about many corner cases > without a clear "regular"? Or are they just few corner cases? One easy-ish way, which would be kind of ugly because coupled to the specific file system capabilities (e.g Btrfs), would be to leverage CoW features and create a Btrfs snapshot at the beginning of the test suite, and reverting to it after it's run. But even that is not fullproof; that'd only protect the build directory, say, not the store location (some check phases are moved after installation). I agree it looks tricky to get it right (and even trickier to prove/test for it) :-). I think the lower fruits are in looking at making the test suite of the few common offenders more robust (using libfaketime or the likes) to prevent (re)occurrences of time bombs in the future. -- Thanks, Maxim
Re: Divvying up service definitions
On Tue, Nov 07, 2023 at 10:56:11AM -0500, Maxim Cournoyer wrote: > Hi Bruno, > > Bruno Victal writes: > > > Hi, > > > > As the gnu/services and gnu/home/services grow, I think we should > > consider divvying the services into stand-alone modules or > > subdirectories. > > > > Consider the ⌜dovecot-service-type⌝ in gnu/services/mail.scm: as of > > commit 'd22d2a05c389207f8cdcf824be7738b1499a987c' this service > > definition is nearly 1600 lines long, with the remainder of the file > > comprising of four other services with rudimentary support. > > > > It becomes troublesome working with such amalgamations as it makes it > > hard to keep track of the used modules and bindings, especially when > > define-configuration is used since the serializing procedures might be > > used by various service definitions. Further complicating things is > > 'define-maybe', whose use monopolizes the predicate and serializers for > > a particular service definition. > > > > Now, I'm not saying that we should go and split everything into its own > > module, I'm saying that we should be allowed to split some of them if > > convenient (all subjective but I believe we can see that working with a > > monolithic file in the kilolines where the interactions aren't obvious > > is not fun, and that's without bringing in the hygienic issues > > surrounding define-configuration and define-maybe). > > > > Some considerations (using dovecot-service-type as an example): > > * Splitting this as gnu/services/mail/dovecot.scm. > > We preserve the logical grouping of the services (with the addition > > that, for extremely comprehensive definitions, these can be neatly > > organized into subdirectories. (same structure seen with gnu/*.scm) > > A drawback is that 'use-service-modules' might not work with this > > although I wonder whether 'use-service-modules' & co. provide any > > value if we are already doing '(use-modules (gnu) …)' to begin with. > > They look redundant IMO. > > > > * Splitting this as gnu/services/dovecot.scm. > > We keep it compatible with 'use-service-modules' at the cost of having > > a multitude of files under gnu/services, without any logical grouping > > (messy). > > That's a great initiative! I agree that multiple 'define-configuration' > services per file can be a bit messy, having to use prefixes everywhere, > making the definitions more verbose. > > I don't have a strong preference of the caterogization of services, but > would perhaps prefer the first one (gnu/services/mail/dovecot.scm), > which could then make it easy to offer some interface as > gnu/services/mail.scm that'd re-export all that is needed (would that > work, or reintroduce the same top-level clashes?). I assume the define-maybe's aren't public, so I'd guess that shouldn't cause a problem as long as they aren't exported. There's some services, like ntpd and openntpd, which reuse the service user/group between them, I think with those being intentional about making sure there aren't clashes, or making sure they line up, would also be a good choice. Or moving the define-maybes to the top of the file and reusing them between services. Is that a possibility? -- Efraim Flashner רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: Turning off tests leads to a different store item
Hi, On Wed, 08 Nov 2023 at 22:17, Maxim Cournoyer wrote: > I agree it looks tricky to get it right (and even trickier to prove/test > for it) :-). Yeah. I have tried a rough “proof-of-concept” i.e., two derivations: one which deletes ’check’ phase and the other which deletes ’build’ phase and depends on the former. Re: Turning off tests leads to a different store item Simon Tournier Thu, 02 Nov 2023 18:02:18 +0100 id:86y1fgm6lh@gmail.com https://lists.gnu.org/archive/html/help-guix/2023-11 https://yhetil.org/guix/86y1fgm6lh@gmail.com In this simple case, it works. :-) But to have something robust, IMHO, it would probably mean 1. create other objects (record) different from and revamp the build systems. And then 2. rebuild many packages for 3. evaluate the ratio between the number of packages that works this way vs the number of packages that fails this way. Oof! That’s a fun project… :-) > I think the lower fruits are in looking at making the test suite of the > few common offenders more robust (using libfaketime or the likes) to > prevent (re)occurrences of time bombs in the future. I agree. On a side note, one of the issue is the time of some tests. Sometimes, packaging is frustrating: build takes ages, then you fix some tests, think it will be good, re-launch “guix build”, another test failing, repeat. It could nice to be able to cache the result of the build phase. Cheers, simon
Re: Better support remote deployment
On Wed, Nov 08, 2023 at 09:36:46AM -0500, Maxim Cournoyer wrote: > Hello, > > Felix Lechner writes: > > > Hi, > > > > On Tue, Nov 07 2023, Maxim Cournoyer wrote: > > > >> Then your suggestion to extend 'guix package' to be able to > >> install a profile from a store profile sounds useful, or perhaps a new > >> 'guix deploy-profile' (or a better name / sub-command?) that would > >> automate the copy and installation in one step. > > > > We have a lot of sub-commands already. Could 'guix deploy' instead be > > fed something other than a 'machine' record? > > Perhaps too much of an overload of that command; not sure. > > > How about a 'remote-user-profile'? > > 'guix package' is already the command we use to create profiles; maybe > it could accept a '--remote' argument to operate on a remote machine? > Or is this not what 'GUIX_DAEMON_SOCKET=ssh://your-target guix package > -m manifest.scm' can already accomplish? :-) Currently there's no ssh connection caching, so using GUIX_DAEMON_SOCKET can be quite slow, I normally allow 5-10 minutes for a profile. That works best when you can't use guix copy to send derivations over. The profile effectively already exists, it doesn't need to be computed on each machine. It does seem more like deploy: take this profile, send it to that machine, and activate it there. `guix deploy` already takes a file, the file could have a profile (or home config?) and a location for the profile. And a user. -- Efraim Flashner רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: Recommended way of packaging web themes
On 2023-11-08 15:44:46 -0800, Felix Lechner via Development of GNU Guix and the GNU System distribution. wrote: > Hi Felipe, > > On Wed, Nov 08 2023, Luis Felipe wrote: > > > I'm looking into packaging the current Guix's website theme > > [...] I think it would be easier for other > > Guix websites and services (e.g. CI, QA, DATA) to use them and hopefully > > achieve a more uniform look and feel for the network of Guix web fronts. > > Could you simply publish the resources online (except Javascript, of > course) so any site can use them? If that will be done, please try to provide a versioned, well, version as well, so that subresource integrity can be used. > > It would be similar to Google Fonts. [1] > > Then I would package the resources being served in such a way that > anyone can easily set up a second server. > > Kind regards > Felix > > [1] https://fonts.google.com/ > Tomas -- There are only two hard things in Computer Science: cache invalidation, naming things and off-by-one errors. signature.asc Description: PGP signature
