Re: State of core-updates

[Björn Höfling]

On Wed, 15 Mar 2023 14:33:36 +0100
Andreas Enge  wrote:

> I more or less tried this by building things on berlin; however, big
> packages (?, precisely: openjdk checkout (not even building!)

OpenJDK@9 and @10 builds are fixed now on master.

Björn




pgpqnmLKJ8OMX.pgp
Description: OpenPGP digital signature

Re: State of core-updates

[Andreas Enge]

Am Thu, Mar 16, 2023 at 10:33:08AM +0100 schrieb Björn Höfling:
> Andreas Enge  wrote:
> > I more or less tried this by building things on berlin; however, big
> > packages (?, precisely: openjdk checkout (not even building!)
> OpenJDK@9 and @10 builds are fixed now on master.

Thanks! Actually they did build on my laptop, the problem was only with
offloading. Apparently a new problem appeared later, thanks for the fix!

Am Wed, Mar 15, 2023 at 12:20:28PM -0700 schrieb Felix Lechner:
> On Wed, Mar 15, 2023 at 7:57 AM Andreas Enge  wrote:
> > Somehow the sending of store items when offloading poses problems
> Somehow I think I see a similar problem with 'guix deploy,' which I
> have to restart manually several times in order to complete the store
> transfer. In my case, the error presents as an SSH error ("parent
> process is not connected" I think) though, instead of a Cuirass
> timeout.

It may be the same root cause. I did not suspect anything when big packages
needed to be transferred, but then it keeps happening with small packages
such as bash or imagemagick. And it becomes very annoying with packages
that have many inputs. As soon as I see no new line "exporting path..."
after a few seconds, it is almost certain that something broke and we will
end up with a timeout one hour later. I am trying with dblatex for about
a dozen times now. At some point in time, all machines of the build farm
will end up having (almost) all the inputs :-|

Yes, then building the package took no time at all! :-)

Andreas

Offloading problems on berlin

[Ludovic Courtès]

Hi,

Andreas Enge  skribis:

> exporting path 
> `/gnu/store/grd39lms6g8pg3r1zlvz7a5qwlymybhy-ghc-9.2.5-src.tar.xz'
>
> building of `/gnu/store/i7b6ffy3jp8adanvkj7zbv3mb5cdsxnq-ghc-9.2.5.drv' timed 
> out after 3600 seconds of silence

There’s a problem with offloading on berlin:

  https://issues.guix.gnu.org/61839

I occasionally offload from my laptop to machines at work but didn’t hit
this bug, so we’ll have to see if it’s specific to berlin or not.

Ludo’.

Re: Core-updates and cross-compilation

[Ludovic Courtès]

Hi,

Josselin Poiret  skribis:

> I've been looking at the state of most failures for the CI jobset for
> core-updates, and we have a couple of problems:
>
> - gcc < 9 and gcc == 12 never cross-compile.

I agree with Andreas, I think that’s okay.

[...]

> - we can't build the cross toolchain for the hurd, because the glibc
>   upgrade to 2.35 would require newer gnumach headers, itself with a
>   newer mig.  All these upgrades would be local and pretty ok if they
>   didn't also require a glibc patch to make the configure script of
>   glibc work (right now it would check for presence of headers without
>   -ffreestanding, even though we clearly don't have the glibc built
>   yet!).  This would cause a world-rebuild as well.  I don't know how
>   much work fixing the rest would be, but that's probably the only glibc
>   patch that's needed.

It’s really great that you’ve been looking at this!  If I can be of any
help, of if you feel desperate ;-), please don’t hesitate to ping me on
IRC.

>   Also note that Hurd now seems to have some quite recent git tags,
>   which are also used by Debian, so we can expect less random commit
>   combinations not working.

The good news is that Samuel Thibault is now officially a (actually:
the) Hurd maintainer, which means they should be able to get upload
permission to ftp.gnu.org—a step in the right direction.

> Should we consider these blockers for a core-updates merge?  Should we
> somehow stop supporting the first use-case?

In the past, I’d use ‘etc/release-manifest.scm’ as a way to check for
merge (or release) blockers.

When it comes to cross-compilation, we must be able to cross-compile the
bootstrap tarballs, as in:

  ./pre-inst-env guix build bootstrap-tarballs \
 --target=aarch64-linux-gnu -n

That works well on ‘core-updates’ so far, except for i586-pc-gnu.

Thanks,
Ludo’.

Re: Caching test results separately?

[Ludovic Courtès]

Hello,

Josselin Poiret  skribis:

> As it stands it's really not possible, as
>
> 1) testing is part of the build process itself and
> 2) we can't look-up any stateful info like this from the building
> process (of course!)
>
> But I would really like for tests to move out of build phases, the
> advantages would be twofold: less build time for a lot of packages, and
> better environment management for tests (we could rely on better
> interaction with other packages, more complicated mocking, use linux
> namespaces to their fullest extent, etc.).  That would require a huge
> change to Guix though, so it's more of a dream than anything concrete.

I don’t think that’s really possible in practice.

But… there’s another idea I’ve been willing to work on for some time:
recognizing build results that are identical modulo references, such
that we can replace full rebuilds by mere reference rewriting in some
cases.

Examples of derivations that lead the same result modulo references:
building a package with/without tests, building a package with various
test suite adjustments, building a package with different versions of
Bash or whatever interpreter is used by its build scripts, etc.

I’m kinda stuck by the fact I don’t want to add more C++ code though,
which means we first need to get that daemon in Scheme…

Ludo’.

Re: The 🐑 Shepherd gets a service collection

[Ludovic Courtès]

Hi,

Adam Faiz  skribis:

>> I imagine we could develop more convenient services like this, such
>> as basic command scheduler similar to the ‘at’ command, and a
>> syslogd
>> implementation.  The latter could be nice for a couple of reasons:
>> logging would happen from the start and till the end (an improvement
>> over the external syslogd process), and it could let us provide a nicer
>> user interface to view logs (taking inspiration from that of
>> ‘journalctl’).
>> Thoughts?  Ideas?I don't think a command scheduler should be added
>> to the Shepherd, isn't that what mcron does?
> If mcron has any deficiencies for being used as an `at` command, then that 
> can be improved.

The main limitation of mcron for such thing is that it’s entirely
static: it reads a list of job specs upfront and then goes on to
schedule them.  There’s no communication protocol to talk to it and
add/remove jobs on the fly, which is what ‘at’ would need.

> Regarding syslogd, I think a better approach is to tell the services to send 
> their output to stdout and stderror,
> so that logs don't depend on a separate logging service in the first place.

Yes, but:

  1. Some daemons include syslog support even today, sometimes optional,
 sometimes mandatory.

  2. Syslog is a bit more structured than just stdout/stderr output:
 there are facilities and levels, for instance—see syslog(3);
 syslogd provides interesting filtering capabilities.

> Per-service logging is already implemented in the Shepherd, but could be 
> streamlined to have a default logs directory:
> https://skarnet.org/software/s6/s6-log.html#loggingchain

Interesting read, thanks!

Regarding the default logs directory, there’s /var/log already, or did
you mean something else?

Thanks,
Ludo’.

Re: The 🐑 Shepherd gets a service collection

[Ludovic Courtès]

Hi,

Maxim Cournoyer  skribis:

> Ludovic Courtès  writes:

[...]

>> I imagine we could develop more convenient services like this, such as a
>> basic command scheduler similar to the ‘at’ command, and a syslogd
>> implementation.  The latter could be nice for a couple of reasons:
>> logging would happen from the start and till the end (an improvement
>> over the external syslogd process), and it could let us provide a nicer
>> user interface to view logs (taking inspiration from that of
>> ‘journalctl’).
>>
>> Thoughts?  Ideas?
>
> While I also find the journalctl interface to be convenient, the
> underlying database logs is costly in terms of storage and complexity

Agreed (that’s why I mentioned the user interface of ‘journalctl’, not
the storage mechanism of ‘journald’).

Ludo’.

Re: Preservation of Guix (PoG) report 2023-03-13

[Ludovic Courtès]

Hi Timothy,

Timothy Sample  skribis:

> Allow me to present to you a long-overdue update to the Preservation of
> Guix (PoG) report: .  🎉

Yay, thank you!!

Do you think this could be turned into a Guix System service, with an
eye towards making it run on project infrastructure?

> Over the whole set, 77.1% are known to be safely tucked away in the
> Software Heritage archive.  But it’s actually much better than that.  If
> we only look at the most recent sampled commit (from Sunday the 5th),
> that number becomes 87.4%, which is starting to look pretty good!

Truly impressive!

> I have a few more notes on the report, but I want to put this near the
> top of the message so that people will see it.  :)  I wrote a script
> (see attached) that uses the PoG database to find missing sources on a
> packge-by-package basis.  That is, you can run
>
> guix repl specification-to-swhids.scm pog.db bash

Neat!

> • Subversion support (for TeX-based documentation stuff, I guess)
> • bzip2 support for Disarchive (there are 45 bzip2 tarballs)
> • ZIP support for Disarchive (for the 8 ZIP files)
> • lzip support for Disarchive (or a workaround for ed)
> • Fix some issues (gettext is .tar.gz, but something went wrong)
> • Do something with the static bootstrap binaries

Subversion support should probably be high-priority because TeX Live is
deep down in the dependency graph so if we ever lose it, we’re doomed.

Thanks for the update!

Ludo’.

Re: A minimal rootless podman configuration

[Ludovic Courtès]

Hi,

Brian Cully via "Development of GNU Guix and the GNU System
distribution."  skribis:

> Existing attempts to use podman rootless run into issues with the 
> cgroups file system being mounted by elogind. Since we now have 
> seatd and greetd, we can bypass elogind. Using them, I have 
> finally been able to use rootless podman. Since this is something 
> that comes up in IRC with some regularity, I wanted to share the 
> operating system configuration here.

Nice, well done!


[...]

>;;
>;; not strictly required, but without them podman will resort 
>to
>;; single user mapping.
>;;
>(simple-service 'subuid-subgid etc-service-type
>(list `("subuid"
>,(plain-file "subuid"
> (string-join
>  '("root:65536:65536"
>"test:16777216:65536")
>"\n" 'suffix)))
>  `("subgid"
>,(plain-file "subgid"
> (string-join
>  '("root:65536:65536"
>"test:16777216:65536")
>"\n" 'suffix)

OK; that works because we have ‘newuidmap’ and ‘newgidmap’ in
‘setuid-programs’ by default.

Interesting, thanks for sharing!

Ludo’.

Re: intrinsic vs extrinsic identifier: toward more robustness?

[Ludovic Courtès]

Hi!

Thanks for starting this discussion!

Simon Tournier  skribis:

> For sure, we have to fix the holes and bugs. :-)  However, I am asking
> what we could add for having more robustness on the long term.
>
> It is not affordable, neither wanted, to switch from the current
> extrinsic identification to a complete intrinsic one.  Although it would
> fix many issues. ;-)

Sources (fixed-output derivations) are already content-addressed, by
definition (I prefer “content addressing” over “intrinsic
identification” because that’s a more widely recognized term).

In a way, like Maxime way saying, the URL/URI is just a hint; what
matters it the content hash that appears in the origin.

So it seems to me that the basics are already in place.

What’s missing, both in SWH and in Guix, is the ability to store
multiple hashes.  SWH could certainly store several hashes, computed
using different serialization and hash algorithm combinations.

This is what you suggested at
; it was
also discussed in the thread at
.  It
would be awesome if SWH would store Nar hashes; that would solve all our
problems, as you explained.

The other option—storing multiple hashes for each origin in Guix—doesn’t
sound practical: I can’t imagine packages storing and updating more than
one content hash per package.  That doesn’t sound reasonable.  Plus it
would be a long-term solution and wouldn’t help today.

Thoughts?

Ludo’.

Re: Offloading problems on berlin

[Andreas Enge]

Am Thu, Mar 16, 2023 at 02:50:53PM +0100 schrieb Ludovic Courtès:
> There’s a problem with offloading on berlin:
>   https://issues.guix.gnu.org/61839
> I occasionally offload from my laptop to machines at work but didn’t hit
> this bug, so we’ll have to see if it’s specific to berlin or not.

Thanks for pointing out that it is a known issue!
Is there a commandline program that can be used like gkrellm to check
whether data is flowing out of berlin to the build node?

Andreas

Re: gnu: inetutils: Update to 2.4.

[Development of GNU Guix and the GNU System distribution.]

Hi Maxim,

On Tue, Mar 14, 2023 at 6:11 PM Maxim Cournoyer
 wrote:
>
> Until everybody has a good grasp of the new process, I think sticking to
> the documented one may be best for now

The bug was retitled 'core-updates' per your request. [1]

I didn't mean to set a tone in the discussion and merely hoped to
embrace—by mistake it turns out—an existing consensus. My apologies!

May this action please invite further dialog, as needed.

Kind regards
Felix Lechner

[1] https://issues.guix.gnu.org/62154

Re: Offloading problems on berlin

[Andreas Enge]

Am Thu, Mar 16, 2023 at 09:40:37PM +0100 schrieb Andreas Enge:
> Thanks for pointing out that it is a known issue!

And it happens quite systematically with large packages sent as inputs.
When trying to build openjdk, I get without pause output like this:
   building /gnu/store/gqdzxi9idv4rby2g4j9s6h1m3qfbmjph-openjdk-15.0.9.drv...
   guix offload: sending 2 store items (960 MiB) to '141.80.167.158'...
   exporting path 
`/gnu/store/lil0s7gijwglfbvx4ailf9fqjmrw5jw9-openjdk-15.0.9-checkout'
   exporting path 
`/gnu/store/ri7i5r91p8zj4jnznsjxnh35n99fb8k6-openjdk-14.0.2-jdk'
and then it stops.

After maybe 40 times or so it goes through. I do not know whether this is
because the transfer finally succeeds (probably so), or maybe because the
build ends up on the machine that also built the input and still has it
in its store...

Andreas

Re: Offloading problems on berlin

[Kaelyn]

--- Original Message ---
On Thursday, March 16th, 2023 at 8:40 PM, Andreas Enge  wrote:


> 
> 
> Am Thu, Mar 16, 2023 at 02:50:53PM +0100 schrieb Ludovic Courtès:
> 
> > There’s a problem with offloading on berlin:
> > https://issues.guix.gnu.org/61839
> > I occasionally offload from my laptop to machines at work but didn’t hit
> > this bug, so we’ll have to see if it’s specific to berlin or not.
> 
> 
> Thanks for pointing out that it is a known issue!
> Is there a commandline program that can be used like gkrellm to check
> whether data is flowing out of berlin to the build node?

You may be able to use jnettop to see the network streams (it does need 
superuser privileges to run). Not sure how readable/useful it might be if 
berlin has a lot of active connections, though.

HTH,
Kaelyn

> 
> Andreas

Re: The 🐑 Shepherd gets a service collection

[Adam Faiz]

On 3/16/23 22:14, Ludovic Courtès wrote:

The main limitation of mcron for such thing is that it’s entirely
static: it reads a list of job specs upfront and then goes on to
schedule them.  There’s no communication protocol to talk to it and
add/remove jobs on the fly, which is what ‘at’ would need.

Would it be easier to add dynamic job spec support to mcron than adding a new 
command scheduler?


Regarding syslogd, I think a better approach is to tell the services to send 
their output to stdout and stderror,
so that logs don't depend on a separate logging service in the first place.


Yes, but:

   1. Some daemons include syslog support even today, sometimes optional,
  sometimes mandatory.

   2. Syslog is a bit more structured than just stdout/stderr output:
  there are facilities and levels, for instance—see syslog(3);
  syslogd provides interesting filtering capabilities.


Thanks, it looks like syslog is still important for structured logs.

Are there issues of logs sent to syslog being lost even when the syslogd 
service is specified as a requirement of services that use it?
If not, I think it's not necessary to add a syslogd implementation to the 
shepherd.
 

Per-service logging is already implemented in the Shepherd, but could be 
streamlined to have a default logs directory:
https://skarnet.org/software/s6/s6-log.html#loggingchain


Interesting read, thanks!

Regarding the default logs directory, there’s /var/log already, or did
you mean something else?

I do mean /var/log, I felt like #:log-file in make-forkexec-constructor could 
be improved.
Rather than always having to specify the absolute log file path, #:log-file 
could just be set as #t and would then default to /var/log/$canonical-name of 
the service.
 

Thanks,
Ludo’.