Re: New “ungrafting” branch

[Andreas Enge]

Hello,

Am Mon, Dec 14, 2020 at 04:00:36PM -0500 schrieb Leo Famulari:
> I agree, it's not a violation of the functional packaging model in a
> strict sense.

if I understand correctly, it remains confusing in a non-strict sense:
grafting is functional, but it is not the same function. Precisely, the
function for grafted packages has more arguments, the original input and the
replacement one. And only one of these arguments is visible in the package
definition, the other one enters through the backdoor (it is declared in
the input itself).

Andreas

Re: When substitute download + decompression is CPU-bound

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt  skribis:

> Ludovic Courtès  writes:
>
>> Well, ‘guix publish’ would first need to create multi-member archives,
>> right?
>
> Correct, but it's trivial once the bindings have been implemented.

OK.

>> Also, lzlib (which is what we use) does not implement parallel
>> decompression, AIUI.
>
> Yes it does, multi-member archives is a non-optional part of the Lzip
> specs, and lzlib implemetns all the specs.

Nice.

>> Even if it did, would we be able to take advantage of it?  Currently
>> ‘restore-file’ expects to read an archive stream sequentially.
>
> Yes it works, I just tried this:
>
> cat big-file.lz | plzip -d -o big-file -
>
> Decompression happens in parallel.
>
>> Even if I’m wrong :-), decompression speed would at best be doubled on
>> multi-core machines (wouldn’t help much on low-end ARM devices), and
>> that’s very little compared to the decompression speed achieved by zstd.
>
> Why doubled?  If the archive has more than CORE-NUMBER segments, then
> the decompression duration can be divided by CORE-NUMBER.

My laptop has 4 cores, so at best I’d get a 4x speedup, compared to the
10x speedup with zstd that also comes with much lower resource usage,
etc.

> All that said, I think we should have both:
>
> - Parallel lzip support is the easiest to add at this point.
>   It's the best option for people with low bandwidth.  This can benefit
>   most of the planet I suppose.
>
> - zstd is best for users with high bandwidth (or with slow hardware).
>   We need to write the necessary bindings though, so it will take a bit
>   more time.
>
> Then the users can choose which compression they prefer, mostly
> depending on their hardware and bandwidth.

Would you like to give parallel lzip a try?

Thanks!

Ludo’.

Re: Word order in Guix l10n

[Ludovic Courtès]

Hi!

Julien Lepiller  skribis:

> Even when translating to French, I sometimes feel the need to change
> word order, but I end up finding a slightly unnatural way to preserve
> the order of arguments. I don't have an example at hand though.
>
> I don't know enough about guile to know how best to implement that (or
> if that exists already).

This looks like a real issue.  I’m surprised this isn’t already
addressed though: after all, ‘printf’ format strings have the same
problem, right?  How does everyone else deal with that?

Thanks,
Ludo’.

Re: Finding versions of packages

[Ludovic Courtès]

Hi,

zimoun  skribis:

> On Tue, 15 Dec 2020 at 12:16, Ludovic Courtès  wrote:
>
>> However, doing such composition on a per-package basis and as the
>> default way of composing packages is inefficient and, more importantly,
>> the resulting compositions may not work.  A package written for Python 2
>> may not work with Python 3, and so on.
>
> About inefficiency, I agree.  The number of inferiors should be
> minimized, IMHO.
>
> By “the resulting composition may not work”, you mean that what is
> propagated may be incompatible, right?  For example, bytecode or ABI
> incompatibilities.

Yes, all sort of incompatibilities could arise.

The great thing with having all the package definitions in a single tree
is that we know they were tested to work together well (hopefully).  If
you start composing packages taken from different revisions, you lose
that guarantee.

Ludo’.

Re: When substitute download + decompression is CPU-bound

[Pierre Neidhardt]

Ludovic Courtès  writes:

> My laptop has 4 cores, so at best I’d get a 4x speedup, compared to the
> 10x speedup with zstd that also comes with much lower resource usage,
> etc.

Of course, it's a trade off between high compression and high speed :)

Since there is no universal best option, I think it's best to support both.

> Would you like to give parallel lzip a try?

It shouldn't be too hard for me considering I already have experience
with Lzip, but I can only reasonably do this after FOSDEM, so in 1.5
month from now.

If I forget, please ping me ;)

If there is any taker before that, please go ahead! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Ludovic Courtès]

Hi Magali,

Magali  skribis:

> scheme@(guix-user)> (let* ((repo (repository-open cache))
>  (latest-commit
>   (commit-lookup repo (reference-target (repository-head repo)
>     (let loop ((commit latest-commit)
>    (res (list latest-commit)))
>   (match (commit-parents commit)
>  (() (reverse res))
>  ((head . tail)
>   (loop head (cons head res))
> Segmentation fault (core dumped)

I can reproduce the bug; the C backtrace looks like this:

--8<---cut here---start->8---
(gdb) bt
#0  0x7fabfa9d31ee in git_oidmap_get ()
   from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#1  0x7fabfa98d516 in cache_get () from 
/gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#2  0x7fabfa9cbef7 in git_object_lookup_prefix ()
   from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#3  0x7fac01b8866d in ffi_call_unix64 ()
   from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#4  0x7fac01b86ac0 in ffi_call_int () from 
/gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#5  0x7fac01e54f2e in scm_i_foreign_call (cif_scm=, 
pointer_scm=, 
errno_ret=errno_ret@entry=0x7ffe5fc1f95c, argv=0x7fabfe2e4980) at 
foreign.c:1073
#6  0x7fac01ec3a84 in foreign_call (thread=0x7fac014a8d80, cif=, pointer=)
at vm.c:1282
--8<---cut here---end--->8---

Could it be that, if you keep ‘repo’ in a global variable like in the
example zimoun posted, segfault no longer occurs?

I believe that what happens is a bug in Guile-Git: ‘repo’ is “finalized”
(freed) before Guile inspects the commit objects to print them, and when
Guile gets around to printing those commit objects, they now refer to a
repo that has been freed, hence the crash.

Until the bug is fixed, the workaround is to arrange your code so that
the repository object outlives commit objects.  We can discuss this
further here or on IRC if you want.

Hope this helps!

Ludo’.

Re: Help, and get help reviewing patches this Friday (18th)

[Ludovic Courtès]

Hi!

Christopher Baines  skribis:

> I mentioned in [1] that some "Scheduled and regular collaboration on IRC
> to review patches" might help both get some patches reviewed, but help
> get more people involved in reviewing patches, even if they haven't done
> so before.
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00583.html
>
> The suggestion came to do this on a Friday, and I have some time
> tomorrow (Friday the 18th) that I can set aside to be on IRC and try and
> help.

I’ve done a bit of that earlier today.

Friday’s not over, so you can help!  For example:

  • If you submitted a patch and it hasn’t been reviewed yet, ping
committers on #guix and let’s discuss it live!

  • If you’re a committer, take a look at patches of yours that have
been reviewed but not pushed yet (there are quite a few!).

  • If you’re interested in someone else’s patch, ping people and
provide feedback to help pull it off.

Ludo’.

Re: Word order in Guix l10n

[Arun Isaac]

Hi,

> This looks like a real issue.  I’m surprised this isn’t already
> addressed though: after all, ‘printf’ format strings have the same
> problem, right?  How does everyone else deal with that?

For C's printf format strings, gettext supports special syntax to
specify argument order. See
https://www.gnu.org/software/gettext/manual/html_node/c_002dformat-Flag.html

A German example is provided on that page.

"%2$d Zeichen lang ist die Zeichenkette `%1$s'"

Regards,
Arun


signature.asc
Description: PGP signature

Discussion: How to package rust crates now and in future?

[Hartmut Goebel]

Hi,

I suggest to discuss this with a broader community.

*TIL: Shall rust packages be packaged with #:skip-build #t? Shall tests 
be run for all crates?*

*My vote: skip-build #t, no tests*

Background: I just submitted some patches for some crates, setting 
#:skip-build for those I touched or added. My rational:


1) Building crate "libraries" is of no use. Rust still has no notion of 
"libraries", neither shared not static. it does not even provide any 
means to use "object"-files from another package. All crates will be 
build again and again for each package using it. And you will notice 
that the output of most crates will be almost empty (only exception: if 
the crate build a program).


2) This is what the crates importer does: It sets skip-build for all 
packages it imports as dependencies. It also does not add the 
crate-build-dependencies for these packages. (Please note that while I 
made the crates importer to honor semver versions, this has already been 
prepared in other patches and was not argued about.



Am 17.12.20 um 21:08 schrieb Efraim Flashner:

I'm in favor of building the packages anyway, it serves as a check that
the inputs are actually correct.


When I started packaging crates, I did this too. But then I learned, 
that others do not. So we should define how this should be handled in 
the future - and adjust the importer accordingly.


This might not be possible - as there is another issue in the Rust 
ecosystem: The language is still moving fast.


3) If some packages requires rustc 1.46, while our default rustc is 
still 1.40, we need to add rustc-1.46 as an input to this package and to 
many of it's dependents. (AFAIR the package will even depend on *both* 
version then.) Now if we move on to rustc 1.50, extra care has to be 
taken to remove these dependencies.


Even worse: All packages depending on such a package on will also depend 
on rustc 1.46, and all changes to rustc 1.46 will trigger a rebuild - 
without *any* use.


4) Since (2) building rust packages costs *a lot* of resources: time, 
memory and electrical power. As an example, building sequoia takes about 
20 Minutes on my machine. Most of the time is spend compiling 
dependencies of dependencies. And all these dependencies of dependencies 
will be compiled over and over again.


5) *If* we decide to build dependencies, we should restrict this to 
*one* build. This means: either not run the tests or only do a test 
build. The reason is: When running the tests, all the code, including 
the dependencies, is compiled again with some "test" flag set. This will 
add yet another huge amount of time, memory and electrical power.


To give you some figure: A release and test build for sequoia takes 
about 45 minutes on my machine, requiring 9 GB of space in /tmp. So this 
is double the time if the release build only.


I can't imaging how many hours it would take to rebuild sequoia is one 
of the lower level dependencies changes - which is quite often the case 
in rust.


6) This not only effect berlin, but also every user out there requiring 
a rebuild for some reason. This will lead to a very, very bad user 
experience - practically kicking out users with less powerful equipment.


7) Given the rushing climate crisis, we MUST NOT waste this gigantic 
amount of electrical power. We are in a position of huge impact. If we 
decide to save power, hundreds of Guix users will save power (and 
money). If we decide to waste power, this will multiply by the number of 
Guix users.


*It's our responsibility to protect the earth!**Yes to #:skip-build #t.*

--
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Discussion: How to package rust crates now and in future?

[John Soo]

 Hey Hartmut,  

  
I’m not sure which way I fall here. I think probably keepijg ci on for most 
crates makes sense if we can work instead towards real shared libraries.
  

  
In any case, I would like to propose a working group for rust. Perhaps we can 
meet monthly in jitsi or elsewhere.
  

  
What do you think?
  

  
- John
 

Re: Help, and get help reviewing patches this Friday (18th)

[zimoun]

Hi,

On Fri, 18 Dec 2020 at 17:56, Ludovic Courtès  wrote:

> Friday’s not over, so you can help!  For example:

[...]

>   • If you’re interested in someone else’s patch, ping people and
> provide feedback to help pull it off.

Let try to close the oldest patch bug#20255. ;-)

  

>From my understanding, it is almost there.

All the best,
simon

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Magali]

Hi!

On 18/12/2020 12:34, Ludovic Courtès wrote:
> Hi Magali,
>
> Magali  skribis:
>
>> scheme@(guix-user)> (let* ((repo (repository-open cache))
>>  (latest-commit
>>   (commit-lookup repo (reference-target (repository-head repo)
>>     (let loop ((commit latest-commit)
>>    (res (list latest-commit)))
>>   (match (commit-parents commit)
>>  (() (reverse res))
>>  ((head . tail)
>>   (loop head (cons head res))
>> Segmentation fault (core dumped)
> I can reproduce the bug; the C backtrace looks like this:
>
> --8<---cut here---start->8---
> (gdb) bt
> #0  0x7fabfa9d31ee in git_oidmap_get ()
>from 
> /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #1  0x7fabfa98d516 in cache_get () from 
> /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #2  0x7fabfa9cbef7 in git_object_lookup_prefix ()
>from 
> /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #3  0x7fac01b8866d in ffi_call_unix64 ()
>from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
> #4  0x7fac01b86ac0 in ffi_call_int () from 
> /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
> #5  0x7fac01e54f2e in scm_i_foreign_call (cif_scm=, 
> pointer_scm=, 
> errno_ret=errno_ret@entry=0x7ffe5fc1f95c, argv=0x7fabfe2e4980) at 
> foreign.c:1073
> #6  0x7fac01ec3a84 in foreign_call (thread=0x7fac014a8d80, cif= out>, pointer=)
> at vm.c:1282
> --8<---cut here---end--->8---
>
> Could it be that, if you keep ‘repo’ in a global variable like in the
> example zimoun posted, segfault no longer occurs?

Yes, this seems to work just fine.


> I believe that what happens is a bug in Guile-Git: ‘repo’ is “finalized”
> (freed) before Guile inspects the commit objects to print them, and when
> Guile gets around to printing those commit objects, they now refer to a
> repo that has been freed, hence the crash.
>
> Until the bug is fixed, the workaround is to arrange your code so that
> the repository object outlives commit objects.  We can discuss this
> further here or on IRC if you want.

Nice! Thanks.


> Hope this helps!

It sure did :-)


Magali

Re: Discussion: How to package rust crates now and in future?

[Pjotr Prins]

Hi Hartmut,

We are using Rust from Guix and just a few comments/thoughts:

On Fri, Dec 18, 2020 at 07:08:24PM +0100, Hartmut Goebel wrote:
>Hi,
> 
>I suggest to discuss this with a broader community.
> 
>TIL: Shall rust packages be packaged with #:skip-build #t? Shall tests
>be run for all crates?
> 
>My vote: skip-build #t, no tests
> 
>Background: I just submitted some patches for some crates, setting
>#:skip-build for those I touched or added. My rational:
> 
>1) Building crate "libraries" is of no use. Rust still has no notion of
>"libraries", neither shared not static. it does not even provide any
>means to use "object"-files from another package. All crates will be
>build again and again for each package using it. And you will notice
>that the output of most crates will be almost empty (only exception: if
>the crate build a program).

You probably know this, but for the benefit of others: Rust builds
static binaries from source. That is their 'philosophy', see

  https://rust-cli.github.io/book/tutorial/packaging.html

This goes against the grain of Unix shared libraries. Building all
crates is what Rust does.  Correct me if I am wrong.

They are talking about librification of the language
https://rust-lang.github.io/compiler-team/minutes/design-meeting/2020-03-12-shared-library-for-types/
which may lead to a wider idea of libraries.

>2) This is what the crates importer does: It sets skip-build for all
>packages it imports as dependencies. It also does not add the
>crate-build-dependencies for these packages. (Please note that while I
>made the crates importer to honor semver versions, this has already
>been prepared in other patches and was not argued about.
> 
>Am 17.12.20 um 21:08 schrieb Efraim Flashner:
> 
> I'm in favor of building the packages anyway, it serves as a check that
> the inputs are actually correct.
> 
>When I started packaging crates, I did this too. But then I learned,
>that others do not. So we should define how this should be handled in
>the future - and adjust the importer accordingly.
> 
>This might not be possible - as there is another issue in the  Rust
>ecosystem: The language is still moving fast.
> 
>3) If some packages requires rustc 1.46, while our default rustc is
>still 1.40, we need to add rustc-1.46 as an input to this package and
>to many of it's dependents. (AFAIR the package will even depend on
>*both* version then.) Now if we move on to rustc 1.50, extra care has
>to be taken to remove these dependencies.
> 
>Even worse: All packages depending on such a package on will also
>depend on rustc 1.46, and all changes to rustc 1.46 will trigger a
>rebuild - without *any* use.

The language may still be changing - but I think it has gotten to the
point that they can't change too much without hurting large software
projects. It probably is a bad idea to mix different compilers to
build crates in one software stack. Maybe I am misreading your idea,
but we either have a 1.40 build or a 1.46 build.

>4) Since (2) building rust packages costs *a lot* of resources: time,
>memory and electrical power. As an example, building sequoia takes
>about 20 Minutes on my machine. Most of the time is spend compiling
>dependencies of dependencies. And all these dependencies of
>dependencies will be compiled over and over again.
> 
>5) *If* we decide to build dependencies, we should restrict this to
>*one* build. This means: either not run the tests or only do a test
>build. The reason is: When running the tests, all the code, including
>the dependencies, is compiled again with some "test" flag set. This
>will add yet another huge amount of time, memory and electrical power.
> 
>To give you some figure: A release and test build for sequoia takes
>about 45 minutes on my machine, requiring 9 GB of space in /tmp. So
>this is double the time if the release build only.
> 
>I can't imaging how many hours it would take to rebuild sequoia is one
>of the lower level dependencies changes - which is quite often the case
>in rust.
> 
>6) This not only effect berlin, but also every user out there requiring
>a rebuild for some reason. This will lead to a very, very bad user
>experience - practically kicking out users with less powerful
>equipment.

The Rust user experience is that Rust builds all crates or installs a
single (static) binary. Even for Guix developers, the default is to
use cargo because of the insane number of dependencies it typically
pulls in. We package dependencies when we want to deploy, not to
develop. 

The user here is a guix user, of course. So between compiler versions
we rebuild the stack and then distribute binaries. No rebuilds get
triggered normally. I think the use case you are referring to is users
as developers (right?) who are actually used to waiting for cargo.