Re: repl macro (metacommand?) for guix CLI (sub)commands

[Ludovic Courtès]

Hi!

jgart  skribis:

> This is because each of those "CLI calls" end up running `(exit 0)`
> at the end in some form or another.

Note that Guile’s ‘exit’ throws a ‘quit’ exception, which can be caught.

Ludo’.

Re: guix refresh to a specific version?

[Ludovic Courtès]

Hi,

Hartmut Goebel  skribis:

> I pushed my current working state to wip-import-version. The branch
> also contains some patches from bug56295 (Add some Chicken eggs and
> fix egg importer) and bug56318 (Fix github updater).

Neat!

> Basic ideas
>
>  * Add a keyword argument „version“ to all “latest-X-release” functions
>and eventually pass it on to called functions.

For clarity I would rename the ‘latest’ field of  to
‘import’ (say), likewise for the ‘latest-*-release’ procedure.

> These updaters report an error that they can't update to a specific
> version, if a version is given:
>
> ?bioconductor (cran.scm) -- repo supports latest version only
> ?cran -- repo supports latest version only
>   old version available in
> https://cran.r-project.org/src/contrib/Archive/PACKAGENAME
> -cpan  --- no version, not recursive
> -elm  -- no updater
> -elpa  --- no version
> -gem  --- no version
> -hackage  --- no version
> -minetest  --- no version
> -opam  --- no version
> -sourceforge (gnu-maintenance) -- to complicated
> -json  -- n/a
> -texlive  --- no version, not recursive, no updater

I suspect most of them could be change to fetch a specific version, but
that can come later.

I’ll be monitoring guix-patches for the final version.  :-)

Thanks!

Ludo’.

Re: grafted package and CLI

[Ludovic Courtès]

Hi,

zimoun  skribis:

> Or even, it can be confusing:
>
> $ guix shell curl@7.79.1 -- curl --version
> curl 7.84.0 (x86_64-unknown-linux-gnu) libcurl/7.84.0 GnuTLS/3.7.2 
> zlib/1.2.11 libidn2/2.3.1 nghttp2/1.44.0
> Release-Date: 2022-06-27
> [..]
>
> The issue is not new, e.g., see [1].  I proposed a patch [2] (see below)
> which addresses the issue with “guix show”.
>
> However, it does not address issue with “guix package -A | grep ^curl”;
> and it is potentially not fixable because it uses
> ’fold-available-packages’ which loads the cache (for performance) and
> this cache does not contain the ’replacement’ field – it is not a good
> idea to introduce it, IMHO.

Usually, when the replacement is a different version, we make it public,
so it also shows up in ‘guix package -A’.  It’s just a convention, but
it’s probably good enough?

> diff --git a/guix/ui.scm b/guix/ui.scm
> index 7fbd4c63a2..b6497f5e5c 100644
> --- a/guix/ui.scm
> +++ b/guix/ui.scm
> @@ -1528,9 +1528,18 @@ HYPERLINKS? is true, emit hyperlink escape sequences 
> when appropriate."
>(define (package  (string
> +  (define replacement
> +(package-replacement p))
> +
>;; Note: Don't i18n field names so that people can post-process it.
>(format port "name: ~a~%" (package-name p))
>(format port "version: ~a~%" (package-version p))
> +  (when replacement
> +(unless
> + (string=?
> +  (package-version p)
> +  (package-version replacement))
> + (format port "replacement: ~a~%" (package-version replacement
>(format port "outputs: ~a~%" (string-join (package-outputs p)))
>(format port "systems: ~a~%"
>(string-join (package-transitive-supported-systems p)))

I’m all for it!  If you want you can resent the whole thing produced by
‘git format-patch’, or I can apply it and provide a commit message on
your behalf.

Thanks,
Ludo’.

Re: Julia packages on build farms?

[Ludovic Courtès]

zimoun  skribis:

> (use-modules (guix build-system julia)
>  (guix packages)
>  (guix profiles)
>  (gnu packages))
>
>
> (manifest
>  (map package->manifest-entry
>   (fold-packages
>(lambda (package result)
>  (if (eq? (package-build-system package) julia-build-system)
>  (cons package result)
>  result))
>'(

With a commit from July 3rd, I get good coverage:

--8<---cut here---start->8---
$ guix weather -m /tmp/julia.scm
computing 296 package derivations for x86_64-linux...
looking for 311 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  99.7% substitutes available (310 out of 311)
  at least 300.7 MiB of nars (compressed)
  668.9 MiB on disk (uncompressed)
  0.026 seconds per request (0.9 seconds in total)
  38.6 requests per second

  0.0% (0 out of 1) of the missing items are queued
  at least 1,000 queued builds
  x86_64-linux: 381 (38.1%)
  i686-linux: 341 (34.1%)
  powerpc64le-linux: 257 (25.7%)
  aarch64-linux: 21 (2.1%)
  build rate: 43.80 builds per hour
  aarch64-linux: 11.74 builds per hour
  powerpc64le-linux: 9.46 builds per hour
  i686-linux: 9.55 builds per hour
  x86_64-linux: 13.16 builds per hour
looking for 311 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  98.4% substitutes available (306 out of 311)
  140.3 MiB of nars (compressed)
  665.7 MiB on disk (uncompressed)
  0.007 seconds per request (2.2 seconds in total)
  136.6 requests per second
  (continuous integration information unavailable)
$ guix describe
Generation 221  Jul 03 2022 23:52:07(current)
  guix e069de4
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: e069de452a2c923868f5137421b4b6349c38d754
--8<---cut here---end--->8---

Ludo’.

Re: ’guix package --export-manifest’ using ’@version’?

[Ludovic Courtès]

Hi,

zimoun  skribis:

> $ guix package -p gnu/store/vmn2b3cmfwf250i65rgvkamyr4971q7m-profile \
>--export-manifest
> ;; This "manifest" file can be passed to 'guix package -m' to reproduce
> ;; the content of your profile.  This is "symbolic": it only specifies
> ;; package names.  To reproduce the exact same profile, you also need to
> ;; capture the channels being used, as returned by "guix describe".
> ;; See the "Replicating Guix" section in the manual.
>
> (specifications->manifest
> (list "python@3.8" "python-numpy@1.17"))

[...]

> The export of the versions depending on the state appears to me
> confusing; for newcomers or people from the classical package managers.
> And it provides more work when transitioning from imperative to
> declarative.
>
>
> Therefore, I think ’--export-manifest’ should not report the version and
> it should be stateless.  However, we could add a comment about
> “--export-channels”.

(Nitpick: it *is* stateless, in the sense that it only depends on Guix
itself, not on the state of the machine where it is being run.)

I see what you mean, but I disagree with the conclusion.

‘--export-manifest’ emits a comment (shown above) explaining that the
manifest is symbolic and that one needs channel info to replicate the
exact same environment.

It is a departure from traditional package managers, and admittedly
surprising to newcomers.  However, my take on this is that we should be
very upfront about symbolic vs. exact reproducibility.  We would muddy
the waters if we gave version strings the same importance as in other
tools, when we know that a version string means very little.

Thoughts?

Ludo’.

Re: repl macro (metacommand?) for guix CLI (sub)commands

[bokr]

Hi Ludo',

On +2022-07-07 09:31:34 +0200, Ludovic Courtès wrote:
> Hi!
> 
> jgart  skribis:
> 
> > This is because each of those "CLI calls" end up running `(exit 0)`
> > at the end in some form or another.
> 
> Note that Guile’s ‘exit’ throws a ‘quit’ exception, which can be caught.

Any possible gotchas in throwing 'quit' re flush-close-sync?

> 
> Ludo’.
>
--
Regards,
Bengt Richter

Re: First guix pull is too costly

[Giovanni Biscuolo]

Hello Ludo’ and Simon,

Ludovic Courtès  writes:

[...]

> Does ‘git clone’ do a better job than libgit2/Guile-Git (which is what
> ‘guix pull’ uses) in case it gets disconnected while fetching the repo?
> Is it able to resume when libgit2 isn’t?
>
> (I’d be surprised, but I don’t know.)

AFAIU `git clone` is still not able to resume after a failed connection,
there is a SoC2008 proposal [1] but it seems it was never implemented.

Thanks to this stackoverflow thread:
https://stackoverflow.com/questions/3954852/how-to-complete-a-git-clone-for-a-big-project-on-an-unstable-connection

it seems there are two available workarounds to this problem:

1. Use shallow clone:

git clone --depth=1
git fetch --unshallow

if fetch fails unshallowing due to connection problems, AFAIU the next
"unshallow" will continue fetching more objects until it's done

2. git-bundle [2] some tagged release

The bundle itself is an ordinary file, which you can download any way,
via HTTP/FTP with resume support, via BitTorrent, via rsync, etc.  You
can create clone from a bundle, fix configuration [3], and do further
fetches from a proper git repository.

A Guix repo git bundle for every new release, starting from 1.3.0
(1.4?), could be useful for people with slow or unreliable internet
connection.

Could `guix pull` be extended to automatically do this for the user?

I mean: for each channell, `guix pull` could test if the
"channel-bundle" (see below) is available and download it with a
resumable method, then use the git-bundle file to create the clone and
finally fix the remote url in the git configuration.

To define the "channel-bundle" we could add a "bundle" sexp (gexp?) to
the channels.scm file, like this:

--8<---cut here---start->8---

(list (channel
(name 'guix)
(bundle "https://git.savannah.gnu.org/guix-1.3.0.bundle";)
(url "https://git.savannah.gnu.org/git/guix.git";)
(commit
  "a3d86b341d361530127c36fbfbf77d638df2c1de")
(introduction
  (make-channel-introduction
"9edb3f66fd807b096b48283debdcddccfea34bad"
(openpgp-fingerprint
  "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")

--8<---cut here---end--->8---

WDYT?

Ciao, Gio'

P.S.: I'm sorry I still cannot help with proper patches to `guix pull`,
I know it's too easy to just describe ideas than to implement it in
code!



[1] https://git.wiki.kernel.org/index.php/SoC2008Ideas#Restartable_Clone

[2] https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-bundle.html

[3] that means fix the url in the [remote "origin"]; we can easily
document it in our manual

-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

Re: Building, packaging and updating Guix with confidence

[Zhu Zihao]

Josselin Poiret  writes:

> In the second case, I am in the process of adding a very simple Guile C
> extension to Guix that only requires to wrap a simple libc function.
> The C code itself took approx. 5% of my time on it, while adding the
> magical invocations for the Autotools took 35%, and now testing the
> changes is taking 60%,

If your foreign function use case is very trivial? Why not give Guile
dynamic FFI a try?

> because I need to test that `guix pull` works,
> that the local development setup works, and that the guix package works,
> all with a local change that cannot be authenticated and that also
> cannot be referred to with a git-reference [more on that later] since
> it's entirely local.  This is causing me many headaches, and I don't
> think that even after manually testing this with handwritten hacks so
> that these changes actually do appear everywhere, I will be confident
> enough for them to be merged.

It's possible to use guix channel to test a local guix repo. A short
example here.

```
cat > local-channel.scm << "EOF"
(list (channel
   (inherit %default-guix-channel)
   (url "/home/foo/bar/path/to/local/guix/repo")
   (branch "test-branch")))
EOF   

guix time-machine -C local-channels.scm --disable-authentication -- build hello
```

> c) The `guix pull` way.  This is a bit of an hybrid.  Internally, it
> amounts to eval'ing build-aux/build-self.scm inside of the currently
> running guix, which returns a procedure which, given a source code
> checkout of Guix, returns a derivation that builds the new Guix.  This
> itself relies on (guix self) where most of the building code is.  The
> scheme modules are compiled in an ad-hoc manner (not following
> Makefile.am), and files are included without consulting the Makefile.
> This is why `guix pull` users are not affected by [1].  But then, there
> still is an issue with the guix daemon (and in the future the C
> extensions), which is C code.  Since we're not pretending to know how to
> universally configure C packages, we rely on the guix-daemon package
> defined in gnu/packages/package-management.scm of the future Guix, which
> inherits from the guix package itself, and then builds itself roughly
> following b), meaning the daemon is still out-of-date.

This is somewhat "the bootstrap problem". It's very ideal if we can
describe the build graph in Guix with derivations. But we still need a
daemon first to process derivations. So we need to build daemon without
Guix.

This issue may be solved by rewriting daemon in Guile. If daemon is
written in Guile. We can run it without compilation.

-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao


signature.asc
Description: PGP signature

Re: ’guix package --export-manifest’ using ’@version’?

[zimoun]

Hi,

On Thu, 07 Jul 2022 at 09:57, Ludovic Courtès  wrote:

> (Nitpick: it *is* stateless, in the sense that it only depends on Guix
> itself, not on the state of the machine where it is being run.)

I will not argue about what we call state here. :-)


> ‘--export-manifest’ emits a comment (shown above) explaining that the
> manifest is symbolic and that one needs channel info to replicate the
> exact same environment.

I do not understand what you mean by «symbolic» here.


> It is a departure from traditional package managers, and admittedly
> surprising to newcomers.  However, my take on this is that we should be
> very upfront about symbolic vs. exact reproducibility.  We would muddy
> the waters if we gave version strings the same importance as in other
> tools, when we know that a version string means very little.

I miss about what you disagree because «We would muddy the waters if we
gave version strings the same importance as in other tools, when we know
that a version string means very little.» is the exact root of my
comment.

By returning,

(specifications->manifest
(list "python" "python-numpy"))

or

(specifications->manifest
(list "python@3.8" "python-numpy@1.17"))

depending on the current Guix is just doing that: «muddy the waters».


Instead, I think ’--export-manifest’ should *always* return:

(specifications->manifest
(list "python" "python-numpy"))

without ’@x.y.z’.  Other said, be in agreement with the comment:

;;   This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".

and not sometimes ’package name’ and sometimes ’package name + version’;
as if version string has a special meaning.


To be honest, I do not understand: on one hand, we are advocating that
version string is not enough for reproducibility.  On the other hand, we
output version string with a comment «version string is useless, you
need a channel file for replicating».


Since I am missing a point, could you explain more?


Cheers,
simon

Re: grafted package and CLI

[zimoun]

Hi,

On Thu, 07 Jul 2022 at 10:00, Ludovic Courtès  wrote:

> Usually, when the replacement is a different version, we make it public,
> so it also shows up in ‘guix package -A’.  It’s just a convention, but
> it’s probably good enough?

About “guix package -A”, we cannot do more, so let say it is good
enough. ;-)

About other situation (guix shell, install, etc.), maybe it could useful
to display a warning or something.


Aside, the convention is to make the replacement public for different
versions, so my naive question is: why not hide the replaced version?


Cheers,
simon

Re: [EXT] Please reply: License of your contributions to the blog at guix.gnu.org

[Thompson, David]

Hi,

Sorry, I haven't been paying attention to mailing lists lately.

I agree with CC-BY-SA 4.0 and GFDL 1.3+ dual-licensing for my contributions
to the Guix blog.

- Dave

On Mon, Jul 4, 2022 at 9:56 AM Ludovic Courtès  wrote:

> Hello Danny, David, and Laura!
>
> I hope this message finds you well.
>
> If I’m not mistaken, you’re the last people who have not replied to my
> message below, also visible at:
> .
>
> Could you please take a look and reply ASAP?
>
> Your reply might be as simple as “I agree with CC-BY-SA 4.0 and GFDL
> 1.3+ dual-licensing for my contributions to the Guix blog.”
>
> Thanks in advance!
>
> Ludo’.
>
> Ludovic Courtès  skribis:
>
> >> Hello,
> >>
> >> I am emailing you on behalf of the GNU Guix project because you are the
> >> author or coauthor of one or more articles to the blog at
> >> .
> >>
> >> With a few exceptions, these articles do not have a clear license, which
> >> we would like to fix.  We propose to dual-license all the articles under
> >> CC-BY-SA 4.0 and GFDL version 1.3 or later, with no Invariant Sections,
> >> no Front-Cover Texts, and no Back-Cover Texts.
> >>
> >> Do you agree with the proposed licensing terms for your contributions to
> >> the blog?
> >>
> >> If you do, please reply to this message to say so, keeping
> >> guix-devel@gnu.org Cc’d (if you already replied in the previous thread,
> >> you do not need to reply again).
> >>
> >> If you would prefer different licensing terms, or if you have any
> >> questions, feel free to ask them publicly on guix-devel@gnu.org or
> >> privately with guix-maintain...@gnu.org.
> >>
> >> The clarified license will allow us and others to reuse material in the
> >> manual, cookbook, and in other free cultural works.  See
> >> 
> >> for the initial discussion.
> >>
> >> Thanks in advance!
> >>
> >> Ludo’.
>

Re: Building, packaging and updating Guix with confidence

[Josselin Poiret]

Hello,

Zhu Zihao  writes:

> If your foreign function use case is very trivial? Why not give Guile
> dynamic FFI a try?

That could be another option, but I'd like to have autoconf be able to
detect whether the target supports things like posix_spawn and
getrlimit, which I use in the code.

> It's possible to use guix channel to test a local guix repo. A short
> example here.

Right, but for the example of adding extensions it won't work since
there's a part of `guix pull` that uses the guix package, as well as in
the installer or system-wide Guix, as I outlined before.  The issue [1]
I outlined in the opening mail was an issue that is specific to the guix
package method, so there really isn't a way to uniformly test changes
without knowing the intricacies of the different builds and where they
end up (I do know these, having run into these issues myself before).

> This is somewhat "the bootstrap problem". It's very ideal if we can
> describe the build graph in Guix with derivations. But we still need a
> daemon first to process derivations. So we need to build daemon without
> Guix.

I don't think that's the case, (guix self) relies on a working daemon
connection before anything else, the built daemon will just be a part of
the resulting `guix pull` profile, but won't be used to build the new
Guix (as a matter of fact, the build daemon is built... using the build
daemon!).

> This issue may be solved by rewriting daemon in Guile. If daemon is
> written in Guile. We can run it without compilation.

I don't think this is directly related, although some changes that we
could bring to it would definitely ease what I'm proposing here: having
a way to build things directly without relying on a root-owned daemon
running would make the bootstrapping problem easier to solve.

Best,
-- 
Josselin Poiret

Re: grafted package and CLI

[Ludovic Courtès]

zimoun  skribis:

> Aside, the convention is to make the replacement public for different
> versions, so my naive question is: why not hide the replaced version?

You mean hide with the ‘hidden?’ property?

Good question.  There’s probably little point in exposing the original
(replaced) version, so yes, hiding it makes sense I guess.  Should we
just do that systematically?

Ludo’.

Re: ’guix package --export-manifest’ using ’@version’?

[Ludovic Courtès]

zimoun  skribis:

> On Thu, 07 Jul 2022 at 09:57, Ludovic Courtès  wrote:

[...]

>> It is a departure from traditional package managers, and admittedly
>> surprising to newcomers.  However, my take on this is that we should be
>> very upfront about symbolic vs. exact reproducibility.  We would muddy
>> the waters if we gave version strings the same importance as in other
>> tools, when we know that a version string means very little.
>
> I miss about what you disagree because «We would muddy the waters if we
> gave version strings the same importance as in other tools, when we know
> that a version string means very little.» is the exact root of my
> comment.
>
> By returning,
>
> (specifications->manifest
> (list "python" "python-numpy"))
>
> or
>
> (specifications->manifest
> (list "python@3.8" "python-numpy@1.17"))
>
> depending on the current Guix is just doing that: «muddy the waters».
>
> Instead, I think ’--export-manifest’ should *always* return:
>
> (specifications->manifest
> (list "python" "python-numpy"))
>
> without ’@x.y.z’.  Other said, be in agreement with the comment:

Oh sorry, I had misunderstood your comment.

The “@X.Y” bit is added only when necessary to disambiguate the package
spec.  So with today’s Guix, we have:

--8<---cut here---start->8---
$ guix shell guile --export-manifest
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(specifications->manifest (list "guile"))
$ guix shell guile@2.2 --export-manifest
guix shell: warning: ambiguous package specification `guile@2.2'
guix shell: warning: choosing guile@2.2.7 from gnu/packages/guile.scm:250:2
guix shell: warning: ambiguous package specification `guile@2.2'
guix shell: warning: choosing guile@2.2.7 from gnu/packages/guile.scm:250:2
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(specifications->manifest (list "guile@2.2.7"))
$ guix shell guile@2.0 --export-manifest
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(specifications->manifest (list "guile@2.0"))
--8<---cut here---end--->8---

In these cases, we cannot omit the “@X.Y” bit because the manifest would
then designate something different.

Does that make sense?

Ludo’.

Re: grafted package and CLI

[zimoun]

Hi,

On Thu, 07 Jul 2022 at 17:09, Ludovic Courtès  wrote:

> You mean hide with the ‘hidden?’ property?

I do not know what I mean. ;-)

The replacement could have an ’hidden?’ property or not being
’define-public’.  

> Good question.  There’s probably little point in exposing the original
> (replaced) version, so yes, hiding it makes sense I guess.  Should we
> just do that systematically?

Well, we should follow the same strategy independently of the version
bump.  Systematically hide original (replaced) original version.

Bah I do not know, what other think?


Cheers,
simon

Re: ’guix package --export-manifest’ using ’@version’?

[zimoun]

Hi,

On Thu, 07 Jul 2022 at 17:13, Ludovic Courtès  wrote:

> The “@X.Y” bit is added only when necessary to disambiguate the package
> spec.  So with today’s Guix, we have:

Yes, but it is not what I reported initially [1]; it was:

1: https://yhetil.org/guix/868rp8zosc@gmail.com

--8<---cut here---start->8---
$ guix time-machine --commit=fb32a38db1d3a6d9bc970e14df5be95e59a8ab02 \
   -- package -p gnu/store/vmn2b3cmfwf250i65rgvkamyr4971q7m-profile
   --export-manifest

;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest
(list "python" "python-numpy"))
--8<---cut here---end--->8---

--8<---cut here---start->8---
$ guix package -p gnu/store/vmn2b3cmfwf250i65rgvkamyr4971q7m-profile \
   --export-manifest
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest
(list "python@3.8" "python-numpy@1.17"))
--8<---cut here---end--->8---


However, I am not able to reproduce; because I removed this profile – it
comes from my attempt to reproduce Docker pack.  Other said, note that
the profile is not in /gnu/store/ but extracted from a Docker pack built
with --save-provenance.

Trying other examples, I am not able to see it again.  Hum?!


Sorry for the noise if it is a bug between the keyboard and the chair. ;-)


> In these cases, we cannot omit the “@X.Y” bit because the manifest would
> then designate something different.

I agree.


Sorry for the noise.  The current behaviour is the one we agree and is
expected.  I thought that I have observed another.  Hum?!

Well, time for me to take summer holidays? :-)


Cheers,
simon