Disappearing development installation images

[Ludovic Courtès]

Hello Guix!

We got a couple more reports that installation images at
 would disappear.

After investigating, it turns out we had a periodic job on berlin
removing GC roots for *.iso and *.qcow2—no wonder they’d disappear
earlier than expected.  That job was added at a time where we had disk
space issues that have since been resolved.

I’ve reconfigured berlin with this commit:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=489fc437c7b3aa0af41a40d6090eb4c51ced0028

… and ran ‘herd restart mcron’.  So from now on, those GC roots will
remain until for 30 days, the default value of ‘ttl’ in
‘cuirass-configuration’:

  
https://guix.gnu.org/manual/devel/en/html_node/Continuous-Integration.html#Cuirass

Ludo’.

Re: networking service not starting for a network-route setting (was for network with netlink-response-error errno:17)

[Giovanni Biscuolo]

Hello,

OK I've managed to fix my networking problem, here is how I did it...

Giovanni Biscuolo  writes:

[...]

> The networking service is failing with this message (manually copied
> here, please forgive mistakes):

now that I can connect via SSH, I can copy the actual messages:

--8<---cut here---start->8---

Jun 14 11:28:32 localhost vmunix: [6.258520] shepherd[1]: Starting service 
networking...
Jun 14 11:28:32 localhost vmunix: [6.472949] shepherd[1]: Service 
networking failed to start.
Jun 14 11:28:32 localhost vmunix: [6.474842] shepherd[1]: Exception caught 
while starting networking: (no-such-device "swws-bridge")
Jun 14 11:28:32 localhost vmunix: [6.492344] shepherd[1]: Starting service 
networking...
Jun 14 11:28:32 localhost vmunix: [6.509652] shepherd[1]: Exception caught 
while starting networking: (%exception #<&netlink-response-error errno: 17>)
Jun 14 11:28:32 localhost vmunix: [6.510034] shepherd[1]: Service 
networking failed to start.

--8<---cut here---end--->8---

> The strange thing is that all the configured interfaces: eno1

I truncated the list, the actual list of interfaces was (and is):

--8<---cut here---start->8---

g@ane ~$ ip addre ls
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope global lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: eno1:  mtu 1500 qdisc pfifo_fast state UP 
group default qlen 1000
link/ether b4:2e:99:c5:cc:1c brd ff:ff:ff:ff:ff:ff
inet 162.55.88.253/24 scope global eno1
   valid_lft forever preferred_lft forever
inet6 fe80::b62e:99ff:fec5:cc1c/64 scope link 
   valid_lft forever preferred_lft forever
3: swws-bridge:  mtu 1500 qdisc noqueue state 
UP group default qlen 1000
link/ether 52:54:00:9b:c6:63 brd ff:ff:ff:ff:ff:ff
inet 192.168.133.1/24 brd 192.168.133.255 scope global swws-bridge
   valid_lft forever preferred_lft forever
4: vnet0:  mtu 1500 qdisc noqueue master 
swws-bridge state UNKNOWN group default qlen 1000
link/ether fe:54:00:ff:e2:fd brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:feff:e2fd/64 scope link 
   valid_lft forever preferred_lft forever
5: vnet1:  mtu 1500 qdisc noqueue master 
swws-bridge state UNKNOWN group default qlen 1000
link/ether fe:54:00:41:53:1e brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe41:531e/64 scope link 
   valid_lft forever preferred_lft forever
6: vnet2:  mtu 1500 qdisc noqueue master 
swws-bridge state UNKNOWN group default qlen 1000
link/ether fe:54:00:3d:17:90 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe3d:1790/64 scope link 
   valid_lft forever preferred_lft forever
7: vnet3:  mtu 1500 qdisc noqueue master 
swws-bridge state UNKNOWN group default qlen 1000
link/ether fe:54:00:64:81:8f brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe64:818f/64 scope link 
   valid_lft forever preferred_lft forever

--8<---cut here---end--->8---

> Please find below the relevant parts of the configuration of my host.
>
> As you can see I've installed a libvirt daemon service (it is working)
> with an autostarted (by libvirt) bridge interface named "swws-bridge"

[...]

> --8<---cut here---start->8---

[...]

sorry I missed to add some relevant definitions I have at the start of
my config.scm file:

(define ane-wan-device "eno1")
(define ane-wan-ip4 "162.55.88.253")
(define ane-wan-gateway "162.55.88.193")
(define swws-bridge-name "swws-bridge")

>(list
> (service static-networking-service-type
>(list (static-networking
>   (addresses (list (network-address
> (device ane-wan-device)
> (value (string-append ane-wan-ip4 
> "/24")
>   (routes (list (network-route
>  (destination "default")
>  (gateway ane-wan-gateway))


the next one the problematic part of my static-networking configuration:

> ;; ip route add 10.1.2.0/24 dev 
> swws-bridge via 192.168.133.12
> (network-route
>  (destination "10.1.2.0/24")   ;; 
> lxcbr0 net
>  (device swws-bridge-name)
>  (gateway "192.168.133.12"
> ;; on node002

I've commented out this network-route part and now the networking
service is running fine at boot (and after a restart obviously)

I think that the missing "swws-bridge" interface when the static-netwo

Re: [PATCH Cuirass 1/4] specification: Ensure name is a symbol.

[Ludovic Courtès]

Hi Romain,

I’ve applied the whole series to Cuirass:

  81d2530 * http: Add /admin/gitlab/event.
  10cf59f * gitlab: Add module for Gitlab JSON objects definition.
  1da873b * specification: Ensure name is a symbol.

I took the liberty to merge the commit that adds test with the one that
defines the HTTP endpoint, and I also added a comment at the top of
gitlab.scm giving context.

For those following along: Romain and I work together at Inria.  This
feature will allow us to hook Cuirass into GitLab so that merge requests
on the channels hosted at https://gitlab.inria.fr/guix-hpc trigger
builds on https://guix.bordeaux.inria.fr (technically, it adds a new
“spec” that gets deleted once the merge request is merged).

We still need to figure out how to get the build result back into
GitLab, but that’s a start.

Feedback welcome!

Thanks,
Ludo’.

[PATCH Cuirass 0/4] Add preliminary support for Gitlab webhooks

[Romain GARBAGE]

This patch series adds support for Gitlab webhooks event, more
specifically the merge-request event.

The first patch fixes an ambiguity on the type of the NAME field in the
SPECIFICATION record definition.

Romain GARBAGE (4):
  specification: Ensure name is a symbol.
  gitlab: Add module for Gitlab JSON objects definition.
  http: Add /admin/gitlab/event.
  http: Add tests for Gitlab API.

 Makefile.am   |  1 +
 src/cuirass/gitlab.scm| 95 +++
 src/cuirass/http.scm  | 88 
 src/cuirass/specification.scm | 11 +++-
 tests/database.scm|  2 +-
 tests/http.scm| 34 +
 6 files changed, 219 insertions(+), 12 deletions(-)
 create mode 100644 src/cuirass/gitlab.scm

-- 
2.45.1

[PATCH Cuirass 2/4] gitlab: Add module for Gitlab JSON objects definition.

[Romain GARBAGE]

* Makefile.am: Add src/cuirass/gitlab.scm.
* src/cuirass/gitlab.scm: Add  and  record
types.
(gitlab-merge-request->specification): New variable.
---
 Makefile.am|  1 +
 src/cuirass/gitlab.scm | 95 ++
 2 files changed, 96 insertions(+)
 create mode 100644 src/cuirass/gitlab.scm

diff --git a/Makefile.am b/Makefile.am
index c58bf58..4a066d3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -52,6 +52,7 @@ dist_pkgmodule_DATA = \
   src/cuirass/store.scm\
   src/cuirass/base.scm \
   src/cuirass/database.scm \
+  src/cuirass/gitlab.scm\
   src/cuirass/http.scm \
   src/cuirass/logging.scm  \
   src/cuirass/mail.scm \
diff --git a/src/cuirass/gitlab.scm b/src/cuirass/gitlab.scm
new file mode 100644
index 000..dab76b5
--- /dev/null
+++ b/src/cuirass/gitlab.scm
@@ -0,0 +1,95 @@
+ gitlab.scm -- Gitlab JSON mappings
+;;; Copyright © 2024 Romain Garbage 
+;;;
+;;; This file is part of Cuirass.
+;;;
+;;; Cuirass is free software: you can redistribute it and/or modify
+;;; it under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation, either version 3 of the License, or
+;;; (at your option) any later version.
+;;;
+;;; Cuirass is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with Cuirass.  If not, see .
+
+(define-module (cuirass gitlab)
+  #:use-module (cuirass specification)
+  #:use-module (json)
+  #:use-module (guix channels)
+  #:use-module (ice-9 match)
+  #:export (gitlab-event
+gitlab-event-type
+gitlab-event-value
+json->gitlab-event
+
+gitlab-merge-request
+gitlab-merge-request-action
+gitlab-merge-request-project-name
+json->gitlab-merge-request
+gitlab-merge-request->specification))
+
+(define-json-mapping 
+  make-gitlab-source
+  gitlab-source?
+  json->gitlab-source
+  (repo-url gitlab-source-repo-url "git_http_url")
+  (name gitlab-source-name "name"
+string->symbol))
+
+(define-json-mapping 
+  make-gitlab-merge-request
+  gitlab-merge-request?
+  json->gitlab-merge-request
+  (action  gitlab-merge-request-action "action")
+  (source-branch   gitlab-merge-request-source-branch "source_branch")
+  (source  gitlab-merge-request-source "source"
+   json->gitlab-source))
+
+(define-json-mapping 
+  make-gitlab-event
+  gitlab-event?
+  json->gitlab-event
+  (type  gitlab-event-type "event_type"
+ (lambda (v)
+   (string->symbol
+ (string-map (lambda (c)
+   (if (char=? c #\_)
+   #\-
+   c))
+ v
+  (value gitlab-event-value "object_attributes"
+ (lambda (v)
+   ;; FIXME: properly handle cases using field TYPE defined above.
+   ;; This would need to use something like Guix's define-record-type*.
+   (cond
+((assoc-ref v "merge_status")
+ (json->gitlab-merge-request v))
+(#t #f)
+
+(define (gitlab-merge-request->specification merge-request)
+  "Returns a SPECIFICATION built out of a GITLAB-MERGE-REQUEST."
+  (let* ((source-name (gitlab-source-name
+   (gitlab-merge-request-source merge-request)))
+ (source-branch (gitlab-merge-request-source-branch merge-request))
+ (source-url (gitlab-source-repo-url
+  (gitlab-merge-request-source merge-request)))
+ (spec-name (symbol-append 'gitlab-merge-requests-
+   source-name
+   '-
+   (string->symbol source-branch
+(specification
+ (name spec-name)
+ (build `(channels ,source-name))
+ (channels
+  (cons* (channel
+  (name source-name)
+  (url source-url)
+  (branch source-branch))
+ %default-channels))
+ (priority 1)
+ (period 0)
+ (systems (list "x86_64-linux")
-- 
2.45.1

[PATCH Cuirass 1/4] specification: Ensure name is a symbol.

[Romain GARBAGE]

* src/cuirass/specification.scm (ensure-symbol): New variable.
(specification)[name]: Ensure name is a symbol.
* tests/database.scm: Fix test.
---
 src/cuirass/specification.scm | 11 ++-
 tests/database.scm|  2 +-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/src/cuirass/specification.scm b/src/cuirass/specification.scm
index d62037e..7b237e6 100644
--- a/src/cuirass/specification.scm
+++ b/src/cuirass/specification.scm
@@ -138,10 +138,19 @@
 packages
 manifests))
 
+(define (ensure-symbol x)
+  (if (string? x)
+  (string->symbol x)
+  x))
+
 (define-record-type* 
   specification make-specification
   specification?
-  (name   specification-name) ;symbol
+  (name   specification-name
+  ;; There was a confusion in the documentation regarding
+  ;; the type of this field. For this reason, strings are
+  ;; accepted but silently converted into symbols.
+  (sanitize ensure-symbol)) ;symbol
   (build  specification-build ;symbol for %build-types
   (default 'all))
   (channels   specification-channels ;list of 
diff --git a/tests/database.scm b/tests/database.scm
index c4efb0a..010c139 100644
--- a/tests/database.scm
+++ b/tests/database.scm
@@ -190,7 +190,7 @@ timestamp, checkouttime, evaltime) VALUES ('guix', 0, 0, 0, 
0);")
   (let* ((spec (db-get-specification "guix"))
  (channels (specification-channels spec))
  (build-outputs (specification-build-outputs spec)))
-(and (string=? (specification-name spec) "guix")
+(and (eq? (specification-name spec) 'guix)
  (equal? (map channel-name channels) '(guix my-channel))
  (equal? (map build-output-job build-outputs) '("job"))
 
-- 
2.45.1

[PATCH Cuirass 4/4] http: Add tests for Gitlab API.

[Romain GARBAGE]

* tests/http.scm (http-post-json, mr-json-open, mr-json-close mr-spec): New
variables.
New tests for Gitlab API.
---
 tests/http.scm | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/tests/http.scm b/tests/http.scm
index a124761..3281bcc 100644
--- a/tests/http.scm
+++ b/tests/http.scm
@@ -21,6 +21,7 @@
 
 (use-modules (cuirass http)
  (cuirass database)
+ (cuirass gitlab)
  (cuirass specification)
  (cuirass utils)
  (tests common)
@@ -41,6 +42,9 @@
   (call-with-values (lambda () (http-get uri))
 (lambda (response body) body)))
 
+(define (http-post-json uri body)
+  (http-post uri #:body body #:headers '((content-type application/json
+
 (define (wait-until-ready port)
   ;; Wait until the server is accepting connections.
   (let ((conn (socket PF_INET SOCK_STREAM 0)))
@@ -85,6 +89,17 @@
  (channel . "packages")
  (directory . "dir2")))
 
+(define mr-json-open
+  
"{\"event_type\":\"merge_request\",\"object_attributes\":{\"action\":\"open\",\"merge_status\":\"can_be_merged\",\"source_branch\":\"test-branch\",\"source\":{\"git_http_url\":\"https://gitlab.instance.test/source-repo/fork-name.git\",\"name\":\"test-project\"}}}";)
+
+(define mr-json-close
+  
"{\"event_type\":\"merge_request\",\"object_attributes\":{\"action\":\"close\",\"merge_status\":\"can_be_merged\",\"source_branch\":\"test-branch\",\"source\":{\"git_http_url\":\"https://gitlab.instance.test/source-repo/fork-name.git\",\"name\":\"test-project\"}}}";)
+
+(define mr-spec
+  (gitlab-merge-request->specification
+   (gitlab-event-value
+(pk 'json (json->gitlab-event mr-json-open)
+
 (test-group-with-cleanup "http"
   (test-assert "db-init"
 (begin
@@ -323,6 +338,25 @@
 (http-get-body
  (test-cuirass-uri 
"/api/jobs/history?spec=guix&names=fake-job&nr=10"))
 
+  (test-equal "/admin/gitlab/event creates a spec from a new merge request"
+(specification-name mr-spec)
+(begin
+  (http-post-json (test-cuirass-uri "/admin/gitlab/event") mr-json-open)
+  (specification-name (db-get-specification (specification-name 
mr-spec)
+
+  (test-equal "/admin/gitlab/event error when a merge request has already been 
created"
+400
+(response-code (http-post-json (test-cuirass-uri "/admin/gitlab/event") 
mr-json-open)))
+
+  (test-assert "/admin/gitlab/event removes a spec from a closed merge request"
+(begin
+  (http-post-json (test-cuirass-uri "/admin/gitlab/event") mr-json-close)
+  (not (db-get-specification (specification-name mr-spec)
+
+  (test-equal "/admin/gitlab/event error when a merge request has already been 
closed"
+404
+(response-code (http-post-json (test-cuirass-uri "/admin/gitlab/event") 
mr-json-close)))
+
   (test-assert "db-close"
 (begin
   (db-close (%db))
-- 
2.45.1

Re: Disappearing development installation images

[Andreas Enge]

Am Fri, Jun 14, 2024 at 11:58:06AM +0200 schrieb Ludovic Courtès:
> After investigating, it turns out we had a periodic job on berlin
> removing GC roots for *.iso and *.qcow2—no wonder they’d disappear
> earlier than expected.  That job was added at a time where we had disk
> space issues that have since been resolved.
> I’ve reconfigured berlin with this commit:

Excellent! This was indeed a source of bad publicity for the project...
I see that a new ISO image has appeared in the download section of the
website. Thanks!

Andreas