Hi!
Janneke Nieuwenhuizen <jann...@gnu.org> skribis:
>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
> Message-ID:
> <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.jann...@gnu.org>
> From: Janneke Nieuwenhuizen <jann...@gnu.org>
> Date: Thu, 22 Jun 2023 08:30:25 +0200
> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
> time.
>
> Similar to split build of make-go in Makefile.am, this breaks-up building
> directories into chunks of max 25 files. Also force garbage collection.
The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
for each chunk of files: each process starts with an empty heap, which
is not the case here as we reuse the same process.
However, (guix self) is already splitting gnu/packages/*.scm in two
pieces: ‘guix-packages-base’ and ‘guix-packages’. The former is the
closure of (gnu packages base), and the latter contains the remaining
files. Unfortunately this is uneven:
--8<---------------cut here---------------start------------->8---
$ readlink -f $(type -P guix)
/gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep
packages-base
/gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
/gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
/gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
$ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -type f
|wc -l
361
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep
packages$
/gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
ludo@ribbon ~/src/guix [env]$ find
/gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
64
$ guix describe
Generation 271 Aug 20 2023 23:48:59 (current)
guix a0f5885
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05
--8<---------------cut here---------------end--------------->8---
Maxime Devos looked into this a while back:
https://issues.guix.gnu.org/54539
> * guix/self.scm (compiled-modules)[process-directory]: Split building of
> directories into chunks of max 25 files.
[...]
> (define (process-directory directory files output)
> - ;; Hide compilation warnings.
> - (parameterize ((current-warning-port (%make-void-port "w")))
> - (compile-files directory #$output files
> - #:workers (parallel-job-count)
> - #:report-load report-load
> - #:report-compilation report-compilation)))
> + (let* ((size 25) ;compile max 25 files a
> time
> + (chunks (unfold
> + (lambda (seed) (< (length seed) size)) ;p
> + (cute take <> size) ;f
> + (cute drop <> size) ;g
> + files ;seed
> + list))) ;tail
> + (for-each
> + (lambda (chunck)
s/chunck/chunk/
Can you confirm that this reduces memory usage observably? One way to
check that would be to print (gc-stats) from ‘process-directory’, with
and without the change. Could you give it a try?
Intuitively, I don’t see why it would eat less memory; maybe peak memory
usage is lower because we do less at once?
Also, I think we should remove the explicit (gc) call: it should not be
necessary, and if we depend on that, something’s wrong.
Anyhow, thanks for tackling this issue!
Ludo’.
