Re: substitute derivation: also substitute grafts?

[zimoun]

Hi,

On Wed, 05 Oct 2022 at 12:03, Ludovic Courtès  wrote:
> Ricardo Wurmus  skribis:
>
>> In my scenario I have two machines, one building stuff the other only
>> substituting.  The serving machine would be the one deciding whether to
>> enforce substitutability or not.
>
> An is it too expensive for that machine to build grafts locally?

Using my old desktop from work, yes the experience is really poor;
especially for some packages where the number of grafts is sometimes
something, e.g., see [1] where grafting were longer than building.

1: 


Cheers,
simon

Re: Planning for a release, for real

[Christopher Baines]

Ludovic Courtès  writes:

> We need to plan and coordinate.  Releases have to be a group effort;
> some of the most important work won’t be coding but coordination.
> Coordination is key.  I don’t think I should be spearheading that
> effort, but I’m happy to be part of it.
>
> Who’s ready to commit time towards that goal for the coming weeks?
>
> Here’s a list of things to do to get there:
>
>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
> a couple of weeks ago, but then I lost track of it.  Marius?
>
> We need a ‘staging’ champion to keep track of what’s left to be
> done, reports progress, pings people, etc.  That person does not
> have to be hacking like crazy, on the contrary!

I'd like to get qa.guix.gnu.org to the point where it's useful for
getting branches merged. Currently, it's possible to submit builds for
branches, which is happening currently for staging.

While that's great, the substitute availability for the branch is still
poor [1], the builds are happening, but there's just not enough hardware
behind bordeaux.guix.gnu.org currently to keep up with both the large
number of builds on the master branch as well as building staging
quickly.

1: 
http://data.qa.guix.gnu.org/repository/2/branch/staging/latest-processed-revision/package-substitute-availability

I know that's not what you're asking for here, but I think a big problem
when it comes to merging branches is that of checking what's broken, and
that's what I'd like to make easier.

>   • Get base binaries on all supported architectures in a timely
> fashion, or drop some of the architectures.
>
> Namely, ‘make assert-binaries-available’ is currently failing.  It
> uses a manifest that encodes what we consider to be the basic
> requirements for each architecture; it’s not demanding for
> aarch64-linux, even less for armhf-linux and i586-gnu—yet we’re not
> meeting these criteria yet.
>
> We need to look at missing substitutes, address build issues and
> build farm issues that cause them until we get to zero failures.  If
> after some effort we fail to get to zero, then we should consider
> dropping architectures (I’m looking at armhf-linux and i586-gnu
> specifically).
>
> Again we need a champion to keep track of this and ping people so we
> make progress!

Is there a reason why ‘make assert-binaries-available’ is just checking
ci.guix.gnu.org? One of the main reasons why bordeaux.guix.gnu.org
exists is to address the lack of substitutes, and if you do check the
mainifest against bordeaux.guix.gnu.org you do get a slightly higher
percentage.

i586-gnu is a shared problem though, I'll try and see if I can get some
childhurd VMs working, although I'm not sure this'll be timely enough
for the upcoming release.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: system/package testing?

[Maxime Devos]

On 06-10-2022 20:34, Maxime Devos wrote:

  -- I miss


Nevermind this, just some writing error during composing the message


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: debbugs-guix.el helper function

[Ludovic Courtès]

zimoun  skribis:

> In the same spirit, I have:

[...]

> (defun-bug->url my/guix-issues "http://issues.guix.gnu.org/issue/";
>   "Add URL of bug NUMBER to `kill-ring'.")
> (defun-bug->url my/guix-debbugs 
> "https://debbugs.gnu.org/cgi/bugreport.cgi?bug=";
>   "Add (old) URL of bug NUMBER to `kill-ring'.")

I have something similar that I find extremely useful: hitting C-w on a
bug adds the mumi and debbugs URLs to the kill ring.

--8<---cut here---start->8---
(defun ludo-copy-debbugs-url ()
  "Add to the kill ring the URL of the Debbugs issue at point."
  (interactive)
  (let ((url1 (concat "https://bugs.gnu.org/";
  (number-to-string (debbugs-gnu-current-id
(url2 (concat "https://issues.guix.gnu.org/";
  (number-to-string (debbugs-gnu-current-id)
(kill-new url1)
(kill-new url2)
(message "Copied %s and %s" url1 url2)))

(define-key debbugs-gnu-mode-map (kbd "C-w") 'ludo-copy-debbugs-url)
--8<---cut here---end--->8---

Ludo’.

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Julien Lepiller  skribis:

> I'll take care of the cranslations (notifying translators, ensuring string 
> freeze is respected, …)

Great, thanks.

> We need to be careful not to start the stsing freeze step too early. Last 
> time (or previous?) we started a week before the scheduled release date, but 
> the schedule slipped by a few weeks and we had some pressure in the pipeline 
> because some patches could not be applied because of string changes.
>
> Let's try to have a better vision on the planning this time :)

Right!  How long do you think we need for the string freeze?

I feel that thanks to Weblate translations are continuously updated, so
maybe things can be frozen for a shorter period of time?

Thanks,
Ludo’.

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Maxime Devos  skribis:

> I have some unapplied security patches (from before the latest release
> (1.3.0) (!)) (more precisely, some patches that prepare for actually
> being able to write the security patches, once the preparation patches
> are merged, the actual security fixes should be relatively simple); I
> think it would be rather bad for known security fixes to not be
> applied for a whole release, especially given how long release cycles
> are in Guix.

Could you provide links to these?

If they depend on ‘openat’ in Guile, that won’t be an option
unfortunately since we’re not going to upgrade the ‘guile’ package
during that time frame.

Thanks,
Ludo’.

Re: Planning for a release, for real

[Maxime Devos]

On 07-10-2022 11:50, Ludovic Courtès wrote:

Hi,

Maxime Devos  skribis:


I have some unapplied security patches (from before the latest release
(1.3.0) (!)) (more precisely, some patches that prepare for actually
being able to write the security patches, once the preparation patches
are merged, the actual security fixes should be relatively simple); I
think it would be rather bad for known security fixes to not be
applied for a whole release, especially given how long release cycles
are in Guix.


Could you provide links to these?


'These' are the ‘openat’ patches:


If they depend on ‘openat’ in Guile, that won’t be an option
unfortunately since we’re not going to upgrade the ‘guile’ package
during that time frame.


However, upgrading the Guile package is not required, I proposed making 
a separate ‘guile-with-openat’ package (= current Guile + some patches), 
which can then be used by the service code: 
 (without world rebuilds).


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: Supported architectures

[Ludovic Courtès]

Hi,

Efraim Flashner  skribis:

> I'd like to comment on armhf for a moment. My memory is a but rusty, but
> I'm pretty sure that in December of 2021 mesa was bumped from 21.2.x to
> 21.3.x, and at that time it stopped building on/for armhf. I noticed in
> May of 2022 (5 months later) and got the build working again. That we
> went 5 months without anyone saying anything in bug reports that mesa
> wasn't building shows that either everyone who is using it is using
> software that doesn't use mesa, or we really don't have any armhf-linux
> users. I'm not advocating dropping the architecture, but it does feel
> like we're already at a best-effort level with it. As far as the pieces
> needed for bootstrapping aarch64 software (go and probably others),
> those get built anyway as needed by aarch64, so there's no worry about
> losing support for those software bits.

We’re definitely on a best-effort basis.  I think we should try and see
if we can make the armhf-linux selection that’s in
‘etc/release-manifests.scm’ build (I think it doesn’t involve Mesa).  If
we realize it’s out of reach, then we mark armhf-linux as in the same
state as mips64el-linux.

> i586-gnu: Do we have a mini guide on how to setup a build environment?
> Something like "add the childhurd service and the secrets service, with
> these bits and you're all set to go"? I don't mind poking builds from
> time to time, but I'm not sure about how to set it up.

Set up a childhurd and offload to it:

  
https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-childhurd_002c-offloading

The main issue at the moment is lack of substitutes, which is due to the
fact that childhurds won’t start on the build machines behind ci.guix.
I’m investigating this here:

  https://issues.guix.gnu.org/58320

Hopefully we’ll have a bug fix or a workaround soon.

> aarch64-linux: I tried a while ago to fix a bunch of the failed builds
> on ci.guix.gnu.org and I think I made it worse. Right now there are many
> build failures and pending builds. I might see about canceling some of
> them and then restarting individual builds to try to increase coverage
> again.

Right, currently there’s something wrong going on with aarch64-linux
builds on ci.guix, which Maxim and I have been looking at.  In
particular, some of the build machines are unreachable right now and may
need a kick.

The starting point for someone willing to help is this dashboard:

--8<---cut here---start->8---
$ make assert-binaries-available 
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
computing 400 package derivations for x86_64-linux...
looking for 512 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  91.6% substitutes available (469 out of 512)
  at least 3,699.6 MiB of nars (compressed)
  6,450.3 MiB on disk (uncompressed)
  0.021 seconds per request (1.1 seconds in total)
  47.1 requests per second

  7.0% (3 out of 43) of the missing items are queued
  at least 1,000 queued builds
  powerpc64le-linux: 699 (69.9%)
  aarch64-linux: 301 (30.1%)
  build rate: 11.31 builds per hour
  i686-linux: 2.31 builds per hour
  x86_64-linux: 7.08 builds per hour
  aarch64-linux: 2.45 builds per hour
  powerpc64le-linux: 0.49 builds per hour
  armhf-linux: 0.02 builds per hour

Substitutes are missing for the following items:
  /gnu/store/pjxakb6iqarajwj5xxbl4xibd40zrwzs-gettext-0.21-doc  
 x86_64-linux
  /gnu/store/04wlmb4mlbhm3hwpp7cwzk45cc8wr7bm-gettext-0.21  
 x86_64-linux
  /gnu/store/8xjxp2gmg1l3db1n18iq0y8mgilf7faf-zlib-1.2.11-static
 x86_64-linux
  /gnu/store/xdvgaj90p1lr4wj0lvsn2m0sqcxs2zsk-xf86-video-sis-0.12.0 
 i686-linux
  /gnu/store/9gl116gdip5zkpzijk07j3kh3hkj21np-grub-hybrid-2.06  
 i686-linux
  /gnu/store/55yxbr2fywkd6x3pzqhh0z22cl6kf3s3-libreoffice-7.3.5.2   
 i686-linux
  /gnu/store/v99pba09zv9m9wl3dgnrq4i26fkrragz-xf86-video-vesa-2.5.0 
 x86_64-linux
  /gnu/store/z47kyvcrxxq4xwczwlbnap9kfvvfigib-gnome-user-docs-42.0  
 x86_64-linux
  /gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5  
 i586-gnu
  /gnu/store/8fhdpb76nqy3v22jr33j18i1k07rl5n1-xz-5.2.5-static   
 i586-gnu
  /gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34  
 i586-gnu
  /gnu/store/sa1ay72axmi9g75sll2wy9cqpfldfy40-gcc-toolchain-12.2.0-debug
 i586-gnu
  /gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0  
 i586-gnu
  /gnu/store/dypv1jgfzxlkbpp36z393xbdqal1v591-gcc-toolchain-12.2.0-static   
 i586-gnu
  /gnu/store/7bx9jykip9lc13yn2bck1m4q8ccds1mz-make-4.3-debug
 i586-gnu
  /gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3  
 i586-gnu
  /gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0  

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Christopher Baines  skribis:

> Ludovic Courtès  writes:
>
>> We need to plan and coordinate.  Releases have to be a group effort;
>> some of the most important work won’t be coding but coordination.
>> Coordination is key.  I don’t think I should be spearheading that
>> effort, but I’m happy to be part of it.
>>
>> Who’s ready to commit time towards that goal for the coming weeks?
>>
>> Here’s a list of things to do to get there:
>>
>>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>> a couple of weeks ago, but then I lost track of it.  Marius?
>>
>> We need a ‘staging’ champion to keep track of what’s left to be
>> done, reports progress, pings people, etc.  That person does not
>> have to be hacking like crazy, on the contrary!
>
> I'd like to get qa.guix.gnu.org to the point where it's useful for
> getting branches merged. Currently, it's possible to submit builds for
> branches, which is happening currently for staging.
>
> While that's great, the substitute availability for the branch is still
> poor [1], the builds are happening, but there's just not enough hardware
> behind bordeaux.guix.gnu.org currently to keep up with both the large
> number of builds on the master branch as well as building staging
> quickly.
>
> 1: 
> http://data.qa.guix.gnu.org/repository/2/branch/staging/latest-processed-revision/package-substitute-availability
>
> I know that's not what you're asking for here, but I think a big problem
> when it comes to merging branches is that of checking what's broken, and
> that's what I'd like to make easier.

Yes, definitely!

> Is there a reason why ‘make assert-binaries-available’ is just checking
> ci.guix.gnu.org?

It’s mostly that ‘guix weather --display-missing’ is not really helpful
when passed more than one URL: it reports all the things that are
missing on at least one of the URLs (this is what the comment in
‘Makefile.am’ suggests).

The goal of this command is to check that “basic packages” build and are
available, and we can expect basic packages to be available on both
build farms so I think it’s OK to check just one of them.  (When I first
wrote that manifest, I naively thought we’d be at 100% at any time since
this is really the baseline…)

> i586-gnu is a shared problem though, I'll try and see if I can get some
> childhurd VMs working, although I'm not sure this'll be timely enough
> for the upcoming release.

For i586-gnu the expectation is really low right now: see
‘%base-packages/hurd’ in ‘etc/release-manifest.scm’.

Ludo’.

Re: Planning for a release, for real

[Julien Lepiller]

It depends on the language, I'd say a week is good, maybe less as long as we 
can include a week-end (maybe it's personal, but I find more time to contribute 
in the week-ends).

Le 7 octobre 2022 11:49:21 GMT+02:00, "Ludovic Courtès"  a écrit :
>Hi,
>
>Julien Lepiller  skribis:
>
>> I'll take care of the cranslations (notifying translators, ensuring string 
>> freeze is respected, …)
>
>Great, thanks.
>
>> We need to be careful not to start the stsing freeze step too early. Last 
>> time (or previous?) we started a week before the scheduled release date, but 
>> the schedule slipped by a few weeks and we had some pressure in the pipeline 
>> because some patches could not be applied because of string changes.
>>
>> Let's try to have a better vision on the planning this time :)
>
>Right!  How long do you think we need for the string freeze?
>
>I feel that thanks to Weblate translations are continuously updated, so
>maybe things can be frozen for a shorter period of time?
>
>Thanks,
>Ludo’.

Re: guix import doesn't like newlines anymore :(

[zimoun]

Hi,

On jeu., 15 sept. 2022 at 19:10, jgart  wrote:
> Hi does anyone know why guix import does not insert newlines anymore?

You mean this,

--8<---cut here---start->8---
(define-public rust-snafu-0.7

[...]

(license (list license:expat license:asl2.0
(define-public rust-lexiclean-0.0.1

[...]

(license license:cc0)))
(define-public rust-just-1

[...]

(license license:cc0)))
--8<---cut here---end--->8---

instead of this,

--8<---cut here---start->8---
(define-public rust-snafu-0.7

[...]

(license (list license:expat license:asl2.0

(define-public rust-lexiclean-0.0.1

[...]

(license license:cc0)))

(define-public rust-just-1

[...]

(license license:cc0)))
--8<---cut here---end--->8---


Right?


Cheers,
simon

Re: guix import doesn't like newlines anymore :(

[jgart]

On Fri, 07 Oct 2022 13:24:17 +0200 zimoun  wrote:
> Right?

Yup!

Re: substitute derivation: also substitute grafts?

[Mark H Weaver]

zimoun  writes:

> On Wed, 05 Oct 2022 at 12:03, Ludovic Courtès  wrote:
>> Ricardo Wurmus  skribis:
>>
>>> In my scenario I have two machines, one building stuff the other only
>>> substituting.  The serving machine would be the one deciding whether to
>>> enforce substitutability or not.
>>
>> An is it too expensive for that machine to build grafts locally?
>
> Using my old desktop from work, yes the experience is really poor;
> especially for some packages where the number of grafts is sometimes
> something, e.g., see [1] where grafting were longer than building.
>
> 1: 

I wonder how fast grafting could be if it were rewritten in C.

   Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about .

Packaging docs

[jgart]

Hi,

What's are policy for packaging docs deps?

For example, this package has docs deps:

https://github.com/m-lundberg/simple-pid/blob/master/docs/requirements.txt

Should those be packaged and included also in the Guix package definition?

Is our policy for packaging docs deps documented somewhere?

Re: Packaging docs

[Julien Lepiller]

I don't think there's a specific policy, but our packages tend to enable most 
optional dependencies and docs are always useful, so I'd package doc 
dependencies too.

Le 8 octobre 2022 05:08:34 GMT+02:00, jgart  a écrit :
>Hi,
>
>What's are policy for packaging docs deps?
>
>For example, this package has docs deps:
>
>https://github.com/m-lundberg/simple-pid/blob/master/docs/requirements.txt
>
>Should those be packaged and included also in the Guix package definition?
>
>Is our policy for packaging docs deps documented somewhere?
>