Re: bug#47897: [PATCH] substitutes: Don't cache negative lookups or transient errors.

[Christopher Baines]

Ludovic Courtès  writes:

> Hi,
>
> Christopher Baines  skribis:
>
>>> Now, the penalty it imposes is annoying.  I’ve sometimes found myself
>>> working around it, too (because I knew the server was going to have the
>>> store item sooner than 1h).
>>>
>>> Rather than removing it entirely, I can think of these options:
>>>
>>>   1. Reduce the default negative timeouts.
>>
>> I think reducing it is good, as you say, it's possible to override the
>> default from the server side. Just in case someone wants caching
>> behaviour, it might be worth keeping that functionality at least.
>
> OK, let’s do that.
>
>>>   2. Add an option to ‘guix publish’ (and to the Coordinator?) so they
>>>  send a ‘Cache-Control’ header with the chosen TTL on 404.  That
>>>  way, if the server operator doesn’t mind extra load, they can run
>>>  “guix publish --negative-ttl=0”.
>>
>> That sounds sensible. The Guix Build Coordinator doesn't do any serving,
>> that's left to something else like nginx. For the deployments I maintain
>> though, I don't think I'm setting the relevant headers, but I'll look at
>> changing that.
>
> Cool.
>
>> Going back to the %narinfo-transient-error-ttl, if I'm correct in saying
>> that it's not possible to override that, maybe that should also use the
>> relevant header value if set?
>
> Correct, ‘%narinfo-transient-error-ttl’ cannot be overridden.  We can
> halve it if you think that’s useful, thought when that happens, it means
> something’s wrong with the server (returning 500 or similar).
>
> I’ve sent patches to address this, lemme know what you think!

The patches you've sent look good.


signature.asc
Description: PGP signature

Re: “guix gc”, auto gcroots, cluster deployments

[Ricardo Wurmus]

Ludovic Courtès  writes:


Hi!

Ricardo Wurmus  skribis:


There are two problems here:

1) I don’t think “guix gc --list-dead” (or “--list-live”, or 
more
generally “findRoots” in nix/libstore/gc.cc) should delete 
anything.  It should just list and not clean up.


Maybe ‘findRoots’ could populate the list of stale roots and 
it’d be up

to the caller to decide whether to delete them or not?


Yes, this would be better.  It already does this for links whose 
targets exist but cannot be read.


2) For cluster installations with remote file systems perhaps 
there’s

something else we can do to record gcroots.  We now have this
excursion into unreadable space because we use a symlink, but 
the
start ($localstatedir/gcroots/auto) and endpoints 
(/gnu/store/…) are
both accessible by the daemon.  Since these intermediate 
locations are

tied to user accounts, could we not store them in a per-user
directory?

This problem does not exist for user profiles, because the link 
in
unreadable home directories is not all that important; it 
merely 
points to $localstatedir, which is always readable by the
daemon. Perhaps we could do the same for temporary roots and 
let
*users* decide when to let go of them by giving them a command 
to

erase the important links in $localstatedir.

So instead of having a link from
/gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
/home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
record
/var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
pointing to 
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, and
then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link 
at

that.  Yes, removing
/home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
longer free up the profile for garbage collection, but removing
$(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
would.


Yes, but how would per-user/me/auto/* be cleaned up?


Yeah, that’s an open question.

I get the appeal of having these things be cleaned up 
automatically when the link disappears, but if we added this extra 
layer of indirection for cluster deployments this would become 
manual.


Can we make this configurable perhaps…?  On my cluster 
installation I’d rather have a cron job to erase the stuff in 
per-user/me/auto/* on my own terms, than to have “guix gc” fail to 
resolve links and consider it all garbage.


--
Ricardo

Re: GNU Guix 1.3.0rc2 available for testing!

[Maxim Cournoyer]

Hi Chris!

Sorry for the delayed answer.

Chris Marusich  writes:

> Hi,
>
> Maxim Cournoyer  writes:
>
>> A second RC for the upcoming 1.3.0 release is now available for testing:
>
> Thank you for preparing it!
>
> I tested the binary installation using the guix-install.sh script for
> 1.3.0 (not the rc2 candidate, but the actual 1.3.0 release, which I
> noticed was on the FTP server already).  I tested on powerpc64le-linux
> and found no major issues; it worked as expected.

Good :-).

> I did "guix pull" and "guix build hello".  To my surprise, I received a
> substitute:
>
> marusich@guixtestbed:~$ guix build hello
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 0.1 MB will be downloaded:
>/gnu/store/zcs3cj0mqixwng2ldf92haab2vkpsicb-hello-2.10
> substituting /gnu/store/zcs3cj0mqixwng2ldf92haab2vkpsicb-hello-2.10...
> downloading from 
> https://ci.guix.gnu.org/nar/lzip/zcs3cj0mqixwng2ldf92haab2vkpsicb-hello-2.10 
> ...
>  hello-2.10  52KiB239KiB/s 00:00 [##] 
> 100.0%
> /gnu/store/zcs3cj0mqixwng2ldf92haab2vkpsicb-hello-2.10
>
> I guess something is building powerpc64le-linux substitutes?  I had
> thought no substitutes would be available, but certainly it is not a
> problem if substitutes are being built for powerpc64le-linux already.

The graciously provided POWER9 VM from OSUOSL is not yet hooked to the
CI (as a Cuirass worker), as you can see here [0].  I'm guessing the
little substitutes available are a side-effect of my initial testing
from Berlin directly, which can offload manually started builds to the
offload machines.

> I tried building from source on Debian 10 buster ppc64el.  It succeeded,
> but "make check" reported one test failure.  It was in tests/go.scm:
>
> test-name: go-module->guix-package
> location: /home/marusich/guix-1.3.0/tests/go.scm:254
> source:
> + (test-equal
> +   "go-module->guix-package"
> +   '(package
> +  (name "go-github-com-go-check-check")
> +  (version "0.0.0-20201130134442-10cb98267c6c")
> +  (source
> +(origin
> +  (method git-fetch)
> +  (uri (git-reference
> + (url "https://github.com/go-check/check";)
> + (commit (go-version->git-ref version
> +  (file-name (git-file-name name version))
> +  (sha256
> +(base32
> +  "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
> +  (build-system go-build-system)
> +  (arguments
> +'(#:import-path "github.com/go-check/check"))
> +  (propagated-inputs
> +`(("go-github-com-kr-pretty"
> +   ,go-github-com-kr-pretty)))
> +  (home-page "https://github.com/go-check/check";)
> +  (synopsis "Instructions")
> +  (description
> +"Package check is a rich testing extension for Go's testing 
> package.")
> +  (license license:bsd-2))
> +   (call-with-temporary-directory
> + (lambda (checkout)
> +   (mock ((web client)
> +  http-get
> +  (mock-http-get fixtures-go-check-test))
> + (mock ((guix http-client)
> +http-fetch
> +(mock-http-fetch fixtures-go-check-test))
> +   (mock ((guix git)
> +  update-cached-checkout
> +  (lambda* (url #:key ref)
> +(values
> +  checkout
> +  (nix-base32-string->bytevector
> +
> "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")
> +  #f)))
> + (go-module->guix-package
> +   "github.com/go-check/check")))
> expected-value: (package (name "go-github-com-go-check-check") (version 
> "0.0.0-20201130134442-10cb98267c6c") (source (origin (method git-fetch) (uri 
> (git-reference (url "https://github.com/go-check/check";) (commit 
> (go-version->git-ref version (file-name (git-file-name name version)) 
> (sha256 (base32 "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5" 
> (build-system go-build-system) (arguments (quote (#:import-path 
> "github.com/go-check/check"))) (propagated-inputs (quasiquote 
> (("go-github-com-kr-pretty" (unquote go-github-com-kr-pretty) (home-page 
> "https://github.com/go-check/check";) (synopsis "Instructions") (description 
> "Package check is a rich testing extension for Go's testing package.") 
> (license license:bsd-2))
> actual-value: #f
> actual-error:
> + (wrong-number-of-args
> +   #f
> +   "Wrong number of arguments to ~A"
> +   (#sxml-0nf (input)>)
> +   #f)
> result: FAIL

The go importer depends on a recent version of guile-lib (0.2.7), which
added a new #:strict argument to the HTML parser.  We should probably
skip the test depending on the already available HAVE_GUILE_LIB Automake
conditional, like so:

--8<---cut here---start->8

Re: Free software telemetry and the Guix System

[Mark H Weaver]

Hi,

Bone Baboon  writes:
> What types of telemetry in free software programs are compatible with
> the Guix System?

The relevant text in the GNU FSDG is here:


  "No Malware

   The distro must contain no DRM, no back doors, and no spyware."

Of course, this depends on our understanding of what it means to be
e"spyware".  There might be edge cases where this is not clear, but I
hope we can all agree that _any_ kind of telemetry *must* be disabled by
default.

> This is a general question but Audacity is a current example of a free
> software program that is in the process of introducing telemetry to some
> degree.  It does not look like Audacity has implemented telemetry yet.
> Here are two links that provide further information.
>
> https://github.com/audacity/audacity/pull/835

The opening message of that pull request states:

  "1. Telemetry is strictly optional and disabled by default.  No data
  is shared unless you choose to opt-in and enable telemetry.
   
   2. Telemetry only works in the builds made by GitHub CI from the
  official repo (the telemetry URLs are only defined there).
   
   3. If you are compiling Audacity from source, we will provide a CMake
  option to enable the telemetry code. This option will be turned
  off by default."

and:

   "Just to reiterate, telemetry is completely optional and disabled by
default. We will try to make it as clear as possible exactly what
data is collected if the user chooses to opt-in and enable
telemetry. We will consider adding the fine-grained controls that
some of you have asked for."

If these claims are true, then I think this is quite satisfactory for
our purposes.  I wouldn't even object to enabling the telemetry code via
the CMake build-time option, as long as it's "opt-in", i.e. that each
user must explicitly enable it, and only after being made aware of the
consequences of doing so.

What do you think?

Thanks for raising this issue.

  Regards,
Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about .

RE: Free software telemetry and the Guix System

[Cook, Malcolm]

 
> Bone Baboon  writes:
> > What types of telemetry in free software programs are compatible with
> > the Guix System?
> 
> The relevant text in the GNU FSDG is here:
> 
> 
> "No Malware
> 
> The distro must contain no DRM, no back doors, and no spyware."
> 
> Of course, this depends on our understanding of what it means to be
> e"spyware". There might be edge cases where this is not clear, but I
> hope we can all agree that _any_ kind of telemetry *must* be disabled by
> default.
> 
> > This is a general question but Audacity is a current example of a free
> > software program that is in the process of introducing telemetry to some
> > degree. It does not look like Audacity has implemented telemetry yet.
> > Here are two links that provide further information.
> >
> > https://github.com/audacity/audacity/pull/835
> 
> The opening message of that pull request states:
> 
> "1. Telemetry is strictly optional and disabled by default. No data
> is shared unless you choose to opt-in and enable telemetry.
> 
> 2. Telemetry only works in the builds made by GitHub CI from the
> official repo (the telemetry URLs are only defined there).
> 
> 3. If you are compiling Audacity from source, we will provide a CMake
> option to enable the telemetry code. This option will be turned
> off by default."
> 
> and:
> 
> "Just to reiterate, telemetry is completely optional and disabled by
> default. We will try to make it as clear as possible exactly what
> data is collected if the user chooses to opt-in and enable
> telemetry. We will consider adding the fine-grained controls that
> some of you have asked for."
> 
> If these claims are true, then I think this is quite satisfactory for
> our purposes. I wouldn't even object to enabling the telemetry code via
> the CMake build-time option, as long as it's "opt-in", i.e. that each
> user must explicitly enable it, and only after being made aware of the
> consequences of doing so.
> 
> What do you think?

My 2 cents:  I think the Audacity model is exemplary and your interpretation is 
spot on.  I personally want the option of enabling such telemetry, as it may 
well serve my needs and may also give the developer valuable usage and/or crash 
info which is the least I can provide in return for such a great FOSS app as 
Audacity.

> 
> Thanks for raising this issue.
> 
> Regards,
> Mark
> 

Re: Free software telemetry and the Guix System

[Leo Famulari]

On Fri, May 14, 2021 at 06:55:34PM +, Cook, Malcolm wrote:
> > If these claims are true, then I think this is quite satisfactory for
> > our purposes. I wouldn't even object to enabling the telemetry code via
> > the CMake build-time option, as long as it's "opt-in", i.e. that each
> > user must explicitly enable it, and only after being made aware of the
> > consequences of doing so.
> > 
> > What do you think?
> 
> My 2 cents:  I think the Audacity model is exemplary and your interpretation 
> is spot on.  I personally want the option of enabling such telemetry, as it 
> may well serve my needs and may also give the developer valuable usage and/or 
> crash info which is the least I can provide in return for such a great FOSS 
> app as Audacity.

+1

unifying mcron and shepherd, service woes, improvements

[raingloom]

Just throwing this out there:
mcron is basically a very simple service manager, so, why not just move
all its timer functionality into shepherd (like systemd's timer units)
and then we can have a unified way to manage and debug services.

But also, shepherd debugging is kind of terrible too, at least on Guix,
so we (and if no one else gets on it that probably means I) should do
something about that.
For example, it's pretty nontrivial to figure out how to run a
service's command in isolation. So far I always had to find it in
/proc, which is not great. stracing a service that fails to start is
also rather tricky. The best I could come up with is to run strace in a
loop and pgrep for the PIDs I'm interested in. Same goes for GDB, which
was still bad in general last time I tried it.

Security could also be improved probably. Can we have an OCAP model?
I feel like I don't know enough to say, so this mail is meant more as a
conversation starter.

So, can we do better? What other issues are there with Shepherd? Are
they fundamental design problems or just plain old bugs?

Or is everyone else happy with the current design and it's just me who
can't use Shepherd properly? 😅

Help me create a janet-build-system

[Jack Hill]

Greetings Guix,

I'm working on creating a build system for Janet [0] modules. This is my 
first time working with build systems, and while I have a much better 
understanding than when I started, I still don't understand it fully. I've 
attached my work in progress patches. The first five set the stage by 
updating and improving our Janet package. Patch six adds the build system, 
and patch seven adds an example package, janet-posix-spawn.


[0] https://janet-lang.org

Trying to build janet-posix-spawn results in:

```
jackhill@alperton ~/repos/guix [env]$ ./pre-inst-env guix build 
janet-posix-spawn
The following derivations will be built:
   /gnu/store/y1xpmfaqswcmiwzpc1bgak18rbw2k4ad-janet-posix-spawn-0.0.1.drv
   /gnu/store/w52ndznq3v42xxv4lfrpdlmwgn8r0f2x-module-import-compiled.drv
building 
/gnu/store/w52ndznq3v42xxv4lfrpdlmwgn8r0f2x-module-import-compiled.drv...
Backtrace:
In system/base/compile.scm:
 43:4 19 (call-with-output-file/atomic _ _ _)
In ice-9/boot-9.scm:
  1736:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In system/base/compile.scm:
59:11 17 (_)
   155:11 16 (_ #)
   235:18 15 (read-and-compile # ?)
   183:32 14 (compile-fold (#) ?)
In ice-9/boot-9.scm:
   2806:4 13 (save-module-excursion _)
In language/scheme/compile-tree-il.scm:
31:15 12 (_)
In ice-9/psyntax.scm:
  1241:36 11 (expand-top-sequence _ _ _ #f _ _ _)
  1233:19 10 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
   285:10  9 (parse _ (("placeholder" placeholder)) (()) _ c&e (# #) #)
In ice-9/eval.scm:
   293:34  8 (_ #)
In ice-9/boot-9.scm:
   3380:4  7 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
  2565:24  6 (call-with-deferred-observers _)
  3393:24  5 (_)
   222:29  4 (map1 (((guix build gnu-build-system) #:prefix gnu:) # ?))
   222:29  3 (map1 (((guix build utils)) ((guix utils
   222:17  2 (map1 (((guix utils
   3300:6  1 (resolve-interface (guix utils) #:select _ #:hide _ # _ ?)
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
no code for module (guix utils)
```

What have I done wrong?

More generally, I expect there are many ways I could improve the build 
system. What are they?


Thanks!
JackFrom f229d3863f5e93606db1275dcb30ad4f3092496b Mon Sep 17 00:00:00 2001
From: Jack Hill 
Date: Thu, 13 May 2021 01:09:22 -0400
Subject: [PATCH 1/7] gnu: janet: Update to 1.15.5.

* gnu/packages/lisp.scm (janet): Update to 1.15.5.
---
 gnu/packages/lisp.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 24babaf469..6e35062561 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -1053,7 +1053,7 @@ including a built-in database engine and a GUI system.")
 (define-public janet
   (package
 (name "janet")
-(version "1.12.2")
+(version "1.15.5")
 (source
  (origin
(method git-fetch)
@@ -1062,7 +1062,7 @@ including a built-in database engine and a GUI system.")
  (commit (string-append "v" version
(file-name (git-file-name name version))
(sha256
-(base32 "0if514zdmbjvvrsa9x5yfvg2b14sz53yaka12g3yhwkq8ls3qk0c"
+(base32 "0gl0p3x0ylj4hxiwjclg1waxq8csadw60qw6pa9h5y4w1bcqffmk"))
 (build-system gnu-build-system)
 (arguments
  `(#:make-flags (list
-- 
2.31.1

From 3aaa244fac08956af7e71f99b3323465d8c58e87 Mon Sep 17 00:00:00 2001
From: Jack Hill 
Date: Thu, 13 May 2021 00:51:52 -0400
Subject: [PATCH 2/7] gnu: janet: Use #:test-target

* gnu/packages/lisp.scm (janet)[arguments]: Add 'test-target' keyword. Remove
customized 'check' phase.
---
 gnu/packages/lisp.scm | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 6e35062561..2e656cdbbc 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -18,6 +18,7 @@
 ;;; Copyright © 2020 Marius Bakke 
 ;;; Copyright © 2020 Zhu Zihao 
 ;;; Copyright © 2021 Sharlatan Hellseher 
+;;; Copyright © 2021 Jack Hill 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1070,12 +1071,10 @@ including a built-in database engine and a GUI system.")
  (string-append "PREFIX=")
  (string-append "CC=" (assoc-ref %build-inputs "gcc")
 "/bin/gcc"))
+   #:test-target "test"
#:phases
(modify-phases %standard-phases
- (delete 'configure)
- (replace 'check
-   (lambda _
- (invoke "make" "test"))
+ (delete 'configure
 (home-page "https://janet-lang.org/";)
 (synopsis "Functional, imperative and embeddable programming language")
 (description
-- 
2.31.1

From 9ab488ac99c3d4be2f0b5d98a037d89c8f18eb59 Mon Sep 17 00:00:00 2001
From: Jack Hill 
Date: Thu, 13 May 2021 00:55:01 -0400
Subject: [PATCH 3/7] gnu: janet: Prefer PREFIX over DESTDIR

* gnu/packages/lisp.scm (janet)[make-flags]: Specify output pat