Re: branch master updated: remote-server: Fix fetch worker crash.

[Mathieu Othacehe]

Hey Ludo,

> I think it’s be safer (less prone to TOCTTOU races) to write it as:
>
>   (catch 'system-error
> (lambda ()
>   (register-gc-roots drv))
> (lambda args
>   (unless (= ENOENT (system-error-errno args)) ;collected in the meantime
> (apply throw args

Oh, indeed! Fixed with 6dcf2f65cea920b9b1c265de3e2b0abe0048a08e.

Thanks,

Mathieu

Re: Desktops on non-x86_64 systems

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer  skribis:

> I've refreshed the branch again, and now there are no performance
> problems with the cross-built rustc.
>
> But Ludovic mentioned that the binary would need to be statically linked
> rather than dynamically linked, and in the case of rustc that
> complicates things because it relies on dynamic linkage for its
> procedural macros, a feature it uses and thus requires to build itself.

Bah, too bad.

> It could perhaps work to 'guix pack' it into a relocatable pack, but
> that'd be fragile and not very clean, compared to a statically link
> archive.
>
> The road ahead is
>
> 1. Try to 'cargo expand' the crates that use other proc macros crates,
> and de-register the proc-macros crates from the rust sources.  Rust
> bundles about 40 proc-macros crates.  That's not guaranteed to work
> easily, unfortunately, as 'cargo expand' is a lossy process and not
> guaranteed to be correct.
>
> 2. Supposing 1 works, it should be possible to build a statically linked
> rust/cargo.
>
> If the above fail or is too difficult to achieve, we could explore a
> 'guix pack'-based solution.

We could try option #2 ‘guix pack -RR’.  However it seems that it’d
still be a looong road before we have a usable bootstrap binary of Rust.

Option #1 seems even trickier (though I next to nothing about Rust).

I think our energy would be better spend on trying the latest mrustc and
the proposed i686 “fixes”¹, or, as a longer-term solution, trying
GCC-Rust.

Anyhow, my (limited) understanding is that there’s no “obvious” solution
in sight, but rather longer-term approaches that need to be tried and
developed.

Thanks for digging this deep into this!

Ludo’.

¹ https://github.com/thepowersgang/mrustc/issues/78#issuecomment-980830551

Re: Desktops on non-x86_64 systems

[Ludovic Courtès]

Hi,

Maxim Cournoyer  skribis:

>> Ludovic Courtès  writes:

[...]

>>> Perhaps we can address all this in several steps:
>>>
>>>   1. apply the librsvg 2.40 hack now so we can merge
>>>  ‘core-updates-frozen’ this week for real;

[...]

> I hear your frustration w.r.t to delays; I don't mind if this stopgap
> solution is implemented *now*, but I'm skeptical that it'd allow GNOME
> to built.

Status update: there’s now ‘librsvg-for-system’ used in a variety of
places, which selects librsvg 2.40 (in C) on non-x86_64 systems:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates-frozen&id=8477a6d743f3068e449402ade739355878242377

It’s used in a number of places, including GTK+, Emacs, Mate, and Xfce:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates-frozen&id=2561f2720f6afdab47991e6430dc8a1215a27bc7
  
https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates-frozen&id=3bd7ce60530ad363e235f458ecbe2bcc9454242a
  
https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates-frozen&id=efbaa5fcc8caeae7228c8f06c13879e7e920f5ea

We can now build Mate and Xfce on i686, with SVG support in GTK+
(without that I suppose they’d be next to unusable because many/most
icons are SVGs these days.)

What we still cannot build is GNOME, and it seems to be out of reach
because gnome → gjs → rust… unless we can use an older GJS or even
Duktape?

It’s a problem because GDM is in ‘%desktop-services’.

Ludo’.

Re: How to compute SWHID? (with Guix/Disarchive)

[Ludovic Courtès]

Hi,

zimoun  skribis:

> Giving a look at Disarchive, I found how to compute Git-based
> serialization hash and somehow serialization methods of "guix hash"
> needs some clearning; considering '--recursive' is 'nar' serialization
> which is a better name.  Anyway, see [1]. :-)

Neat!

> I would like to add SWH-based serialization hash but I do not find if
> a function already does the hard work.  Any pointer?

I think it’s ‘git-hash-directory’ in (disarchive git-hash).

The other day I learned that the Git CLI ignores empty directories, but
the Git format itself has nothing against empty directories.  Thus SWH
serializes in exactly the same way as Git.

(Can you confirm, Timothy?)

> Moreover, I would like to add* a new export format to "guix show"
> using BibTeX format proposed by SWH.  It would help when writing
> paper. ;-)

That’d be fun!  Maybe we need a separate tool set for scientific
authoring?

Thanks,
Ludo’.

Upgrading storage on bordeaux.guix.gnu.org

[Ludovic Courtès]

Hello!

(+Cc: Andreas.)

Ricardo Wurmus  skribis:

> [space is running out on bayfront, so I wrote:]
>
>> Remember that I’ve got three 256G SSDs here that I could send to
>> wherever bayfront now sits.  With LVM or a RAID configuration these
>> could just be added to the storage pool — if bayfront has
>> sufficient slots for three more disks.
>
> You wrote in response:
>
>> Good to know.  In that case we’d need to come up with (1) an updated
>> Guix System config with LVM, and (2) a way to copy the existing
>> store
>> over to the new storage, which sounds tricky if the existing disk is
>> to
>> be kept.
>
> We could first install Guix System with the adjusted bayfront config
> on a separate machine (e.g. on a build node at the MDC), onto a volume
> with LVM (using as many of the SSDs as needed). Copy signing keys etc
> from bayfront.  Then we’d pretty much export/import the bayfront store
> over the network.  Once everything has been copied, we turn off
> bayfront, swap the disks, boot it up again.  If everything works all
> right we add the original disk (and any unused left-over disks) to the
> LVM volume to extend the storage pool.

Sounds like a plan.  But note that there’s the store and there’s the
cached nars, though maybe we can tolerate missing, say, a week or two of
nars.

It would be more convenient to do that with a machine already in the
vicinity of bayfront though, so we can more easily move the disks there
when we’re ready.  Maybe we could use a machine at Inria or the math
institute next door.  Andreas, WDYT?  (We can work out the details
off-list.)

>> (Also I think we’re down to 1.5 person who could go on
>> site. :-/)
>
> Not great :-/

Here’s a call: if you’re in the whereabouts of Bordeaux, France, and
would like to help, please get in touch with Andreas and myself!  We
need to increase our tramway factor.

Cheers,
Ludo’.

Re: Preservation of Guix Report 2021-11-30

[Ludovic Courtès]

zimoun  skribis:

> Where I am surprised is that PoG does not return 'python-scikit-learn' when:
>
> $ guix lint -c archival python-scikit-learn
> gnu/packages/machine-learning.scm:946:5: python-scikit-learn@0.24.2:
> scheduled Software Heritage archival

This is most likely due to .

Ludo’.

Re: Preservation of Guix Report 2021-11-30

[Ludovic Courtès]

Hi!

Timothy Sample  skribis:

> Here’s a new version of the Preservation of Guix Report:
>
> 

The PoG reports are all 404 right now.

> Here’s what I wrote:
>
>>  This version has a breakdown by different origin types.  The good
>>  news is that Git origins are doing very well.  We’ve confirmed that
>>  97.2% of the 9,272 Git origins that we’re tracking are in the SWH
>>  archive.  Most of the progress there is due to zimoun wading through
>>  the missing packages and telling SWH to store them – thanks, zimoun!

Yay!

> That’s still basically true this month, but we have a few more missing
> Git sources.  Actually, we are starting to lose sources!  If you look at
> the graph of commits, you can see a sharp increase in missing sources
> for recent commits.  It looks like a problem on the SWH side.  Visiting
> [1] and selecting “Show all visits”, you can see that the nixguix loader
> has been having trouble loading our “sources.json” recently.
>
> [1] 
> 
>
> I will try and get in touch with SWH about this.  While it’s troubling,
> it certainly is a good confirmation that doing some basic monitoring is
> important!

Yup.

> That’s the bad news.  The good news is I’ve added support for XZ to
> Disarchive (to be officially released in Disarchive soon).  That means
> that we have information about 4K more sources.  We now know the status
> of 80% of our sources.  Unfortunately, 40% of the XZ sources are
> missing!  Most of them are old, as can be seen in this (secret) graph:
>
> 
>
> (The filename format is “{tar-gz,tar-xz,git}-{rel,abs}-hist.svg” if you
> want to see all the secret graphs.)

Can’t wait to see the secret graphs.  :-)

Thanks!

Ludo’.

Re: bug#52284: Partially unifying packages and inferior packages

[Ludovic Courtès]

Hi,

Maxime Devos  skribis:

> (define (inferior-package->package inf)
>   ;; TODO: somehow make sure no inheritance happens on this package
>   (package
> (name (inferior-package-name inf))
> (version (inferior-package-version inf))
> (replacement (and=> (inferior-package-replacement inf) 
> inferior-package->package))
> (source #f) ; TODO
> (build-system inferior-package-build-system)
> (arguments `(#:inferior-package ,inf))
> (synopsis (inferior-package-synopsis inf))
>     (description (inferior-package-description inf))
> (home-page (inferior-package-home-page inf))
> (location (inferior-package-location inf))
> (inputs (map inferior-inputs->inputs (inferior-package-inputs inf)))
> (native-inputs (map inferior-inputs->inputs 
> (inferior-package-native-inputs inf)))
> (propagated-inputs (map inferior-inputs->inputs 
> (inferior-package-propagated-inputs inf)))
> (transitive-propagated-inputs (map inferior-inputs->inputs 
> (inferior-package-transitive-propagated-inputs inf)))
> (native-search-paths (propagated-package-native-search-paths inf))
> (search-paths (propagated-package-search-paths inf))
> (license #f)) ; TODO

That’s a clever hack!

Longer-term, I think it would be nice(r) to use a type hierarchy somehow
so  instances can truly be used anywhere a 
is expected.

Thanks,
Ludo’.

Re: Preservation of Guix Report 2021-11-30

[zimoun]

Hi,

On Mon, 06 Dec 2021 at 14:10, Ludovic Courtès  wrote:
> zimoun  skribis:
>
>> Where I am surprised is that PoG does not return 'python-scikit-learn' when:
>>
>> $ guix lint -c archival python-scikit-learn
>> gnu/packages/machine-learning.scm:946:5: python-scikit-learn@0.24.2:
>> scheduled Software Heritage archival
>
> This is most likely due to .

Yes, that’s the explanation why “guix lint” is always scheduling it. :-)

The very same mechanism is used to fallback.  Therefore, Guix cannot
uses SWH as fallback for ’python-scikit-learn’; for now. ;-)

PoG uses the same API point*, IIUC.  Somehow, «PoG code finds ’foo’» has
to match «Guix code can use SWH as fallback for ’foo’» and the question
using ’python-scikit-learn’ as example: is it the case?

Other said, does this coverage done by PoG code correctly represent what
Guix code can reach?  Looking for corner cases; the devil is in the
details. ;-)


*API: instead of one query by request, PoG sends 1000 queries by request.
Because SWH limits to 100 request per hour, PoG checks for all packages
when Guix requires… bah! :-)


Cheers,
simon

Re: How to compute SWHID? (with Guix/Disarchive)

[Timothy Sample]

Hi,

Ludovic Courtès  writes:

> zimoun  skribis:
>
>> Giving a look at Disarchive, I found how to compute Git-based
>> serialization hash and somehow serialization methods of "guix hash"
>> needs some clearning; considering '--recursive' is 'nar' serialization
>> which is a better name.  Anyway, see [1]. :-)
>
> Neat!
>
>> I would like to add SWH-based serialization hash but I do not find if
>> a function already does the hard work.  Any pointer?
>
> I think it’s ‘git-hash-directory’ in (disarchive git-hash).

That’s the one.  I only know what SWH does for a few cases:

  • directory: Use their version of ‘git-hash-directory’.

  • file: Use their version of ‘git-hash-file’ (resulting in a ID like
“swh:1:cnt:...”).  I don’t know if they ingest regular files like
this, but if they ingested the file through another means, it will
have that ID.

  • git: Read the directory ID from the Git database.  This is
essentially ‘git rev-parse HEAD:’, where the colon at the end tells
Git to get the “tree” (directory) ID rather than the commit ID.
(I’m not sure if guile-git supports this; so far I’ve just been
shelling out to Git.)

  • hg: Use their version of ‘git-hash-directory’ excluding the “.hg”
directory.

In my work, I’ve been strict about keeping the Git directory IDs based
on the Git database (“.git”) rather than computing them using
‘git-hash-directory’.  Since Guix deletes the Git database before
putting a checkout in the store, that option may not be available to you
(unless you download the repository again).  I’m not sure how much of
problem this would be in practice.  There may be a few edge cases with
submodules and “.gitattributes” to watch out for.

My guess is that as it stands, if a repo has a “.gitattributes” file,
running ‘git-hash-directory’ on the checkout will produce a directory ID
that SWH doesn’t have (they will ignore it, but we will include it).  A
corollary of this guess is that our SWH fallback code for Git will fail
for a repo that has a “.gitattributes” file, since we include it in the
nar hash, but SWH will not provide it.  (I say “guess” because this is
based on some stuff I observed when writing that procedure several
months ago – I haven’t verified any of this.  See also
, which is the same problem but with
submodules instead of “.gitattributes”).

Sorry but all I have to offer is doom and gloom on this one.  :(  You
might be able to get ‘git-hash-directory’ to work well enough on the Git
checkouts that Guix puts in the store, but you’ll have to be careful!

> The other day I learned that the Git CLI ignores empty directories, but
> the Git format itself has nothing against empty directories.  Thus SWH
> serializes in exactly the same way as Git.
>
> (Can you confirm, Timothy?)

I can confirm that a Git tree node of the form

4 empty-directory 4b825dc642cb6eb9a060e54bf8d69288fbee4904

theoretically represents an empty directory named “empty-directory”.
The hash is computed like this:

$ printf 'tree 0\0' | sha1sum
4b825dc642cb6eb9a060e54bf8d69288fbee4904  -

I don’t know anything about where Git excludes this or what would happen
if you manually constructed a Git repo with empty directories, though!


-- Tim

Preservation of Guix Report 2021-12-06

[Timothy Sample]

Hi Guix,

This is an update to the preservation of Guix report.  There are no new
commits or fixed-output derivations in this report, but I spent some
time cleaning up the results, and I think the improvements are worth
sharing.  The last report generated a lot of questions.  This one
doesn’t answer all of them, but it’s a big improvement:



Since the last report, I added many more reference categories and moved
them to the database.  The new categories are 'hg', 'svn', 'cvs', 'bzr',
'tar-bz2', 'tar', 'zip', and 'text'.  Of these, only 'tar' and 'text'
are being processed.  The rest are currently unsupported by my scripts.
Moving the categories to the database allows me to make manual
corrections when needed.  It also encouraged me to look through the
references a bit more carefully to track down some of the weirder 'text'
sources (like Bash patches) and fix up some other ones (in the style of
“/tar_gz?download=yes”).

I also made the fetching code more tenacious.  Now it uses the
content-addressed mirrors from Guix and Nix to find regular files, and
will recover “easy” Git references from SWH (“easy” means the commit is
specified).

Between improving the fetching code and adding 'tar' and 'text'
processing, I’ve computed another 2.5K SWHIDs.  We now have SWHIDs for
86% of our fixed-output derivations.  There are only 51 “unknown”
non-recursive Git sources now (the list is attached).

But that’s not all!

The scripts now categorize failures, so we have a better idea of what’s
going on with the remaining 14% “unknown” sources:

  no-ref:13
  disarchive:   863
  fetch:   1262
  bail:3324
  -
  total:   5462

The “bail” category is all the stuff my scripts don’t yet process, like
Mercurial repositories and bzip2 tarballs.

The “fetch” category is everything the scripts couldn’t track down.

The “disarchive” category is all the tarballs Disarchive failed to
process.  An interesting thing here is that most of them are from Cargo.
Long story short: older versions of Cargo used the “miniz”
implementation of DEFLATE (rewritten in Rust) to compress tarballs.
Disarchive doesn’t support this (yet...?).  There are 686
old-Cargo-produced tarballs in the “disarchive” category.

The “no-ref” category covers a few fixed-output derivations used in
bootstrapping that do not come from an origin record.  I will probably
just load them by hand eventually.

(In the future I hope to put some of this in the report itself.)

One last thing to add is that the SWH folks were very quick to fix the
loading error, so the increase in missing sources for recent commits is
now gone.


-- Tim

0nij3ray7nssvq0lzb352wmnab8ffzk7dgff2c68mvjbh1l6
(git-reference
 (url "https://github.com/fdik/libetpan";)
 (commit "210ba2b3b310b8b7a6ee4a4e35e50f7fa379643f"))

03ym14g9qhjqmryr5z065kynqm8yhmvnbs2djl6vp3i9cmqln8cl
(git-reference
 (url "https://git.savannah.gnu.org/git/emacsy.git";)
 (commit "v0.4.1-37-g5f91ee6"))

04c2vqxg31mk15cfrhzrivykis8fmf0m1d8h1qdjdmlfxd4qwaqf
(git-reference
 (url "https://github.com/hboetes/mg";)
 (commit "20210609"))

06plnhi1489wqsag5wgm16hb1xd1a8nbnb9gw7635d3fidxyb0wp
(git-reference
 (url "https://github.com/erlang/otp";)
 (commit "OTP-24.0.2"))

06w86xk7sjl2x2h3z6msn8kpmwj05qdimcym77wzhz5s94dzh1bl
(git-reference
 (url "https://github.com/hboetes/mg";)
 (commit "20180408"))

07xansmhn4l0b9ghzf56vyx8cqg0q01aq3pz5ikx2i19v5f0rc66
(git-reference
 (url "https://github.com/tgvaughan/elpher";)
 (commit "v1.4.6"))

08yca0a0prrnrc7ir7ajd56yxvxpcs4m1k8f5kf273f5whgr7wzw
(git-reference
 (url "https://github.com/ProtonVPN/linux-cli";)
 (commit "v2.2.4"))

0a066f56hnb9znbwnv1blm31j0ysv05n4wzlkli0zgw087c9047x
(git-reference
 (url "https://source.atlas.engineer/public/next";)
 (commit "1.2.0"))

0anmprm63a88kii251rl296v1g4iq62r6n4nssx5jbc0hzkknanz
(git-reference
 (url "https://git.savannah.gnu.org/git/nomad.git/";)
 (commit "0.2.0-alpha-100-g6a565d3"))

0d269474kk1933c55hx4azw3sak5ycfrxkw6ida0sb2cm00kfich
(git-reference
 (url "https://gitlab.savoirfairelinux.com/sflphone/libiax2.git";)
 (commit "0e5980f1d78ce462e2d1ed6bc39ff35c8341f201"))

0f8zr2jxr0v4zcd98zqx99zxdn768vjpzwxsbsd6ss3if405sq2a
(git-reference
 (url "https://github.com/erlang/otp";)
 (commit "OTP-24.0.5"))

0fy4qsss3i3pkq1rpgjds4aipbwlh1dr9hbbf7jn2a1c63kfks0r
(git-reference
 (url "https://git.zx2c4.com/wireguard-go/";)
 (commit "v0.0.20200320"))

0fzhwbpyndwrmxip9zlcwkrr675l5pzwcygi45hv7w1hn39w0hxp
(git-reference
 (url "https://github.com/emacsattic/relative-buffers";)
 (commit "9762fe268e9ff150dcec2e2e45d862d82d5c4008"))

0gv83i5ybj1z3ykbbldjzf7dbfjszp84c0yzrpshj611b9wp0176
(git-reference
 (url "https://github.com/erlang/otp.git";)
 (commit "OTP-21.0.5"))

0ibq30xrf871pkpasi8p9krn0pmd86rsdzb3jqvz3wnp4wa3hl9d
(git-reference
 (url "https://source.atlas.engineer/public/next";)
 (commit "1.3.0"))

0ifnaclsz7w08mc485i3j1kkcpd1m8q5qamckrfwc375ac13xf4g
(git-reference
 (url "git://anongit.kde