bug#30820: Chunked store references in compiled code break grafting (again)

[Ludovic Courtès]

Mark H Weaver  skribis:

> Nothing that we do will ever fix this problem for good.  Don't pretend
> that patching GCC would fix the problem for good.  There are problems in
> other software as well (e.g. in JAR manifests), and we already patched
> GCC once, and it broke some time later without anyone noticing.

My initial message spread some confusion: the GCC patch still works as
before, but there’s always been a corner case that was improperly
handled.

Now, we currently don’t have tests that would allow us to detect
breakage before it bites, which is a problem.

Ludo’.

bug#30820: Chunked store references in compiled code break grafting (again)

[Ludovic Courtès]

Mark H Weaver  skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver  skribis:
>>
>>> l...@gnu.org (Ludovic Courtès) writes:
>>>
 The recently added glibc grafts triggered issues that, in the end, show
 the return of  (“Store references in 8-byte
 chunks in compiled code”).
>>>
>>> I think that we should generalize our reference scanning and grafting
>>> code to support store references broken into pieces, as long as each
>>> piece containing part of the hash is at least 8 bytes long.
>>>
>>> Here's my preliminary proposal:
>>>
>>> (1) The reference scanner should recognize any 8-byte substring of a
>>> hash as a valid reference to that hash.
>>>
>>> (2) To enable reliable grafting of chunked references, we should impose
>>> the following new restrictions: (a) the store prefix must be at
>>> least 6 bytes, (b) grafting can change only the hash, not the
>>> readable part of the store name, and (c) the readable part of the
>>> store name must be at least 6 bytes.
>>>
>>> (3) The grafter should recognize and replace any 8-byte subsequence of
>>> the absolute store file name.
>>
>> I’m quite reluctant because it would add complexity, it will probably
>> slow things down, and yet it may not handle all the cases, as Danny
>> suggests.
>>
>> Mind you, the GCC patches are not perfect either, but they’re relatively
>> easy to deal with (well, so far at least).  In theory we would need
>> similar patches for LLVM and maybe a couple other native compilers,
>> though, which is obviously a downside, although we haven’t had any
>> problems so far.
>
> We would also need to find a solution to the problem described in the
> thread "broken references in jar manifests" on guix-devel started by
> Ricardo, which still has not found a satifactory solution.
>
>   https://lists.gnu.org/archive/html/guix-devel/2018-03/msg6.html
>
> My opinion is that I consider Guix's current expectations for how
> software must store its data on disk to be far too onerous, in cases
> where that data might include a store reference.  I don't see sufficient
> justification for imposing such an onerous requirement on the software
> in Guix.

In practice Guix and Nix have been living fine under these constraints,
and with almost no modifications to upstream software, so it’s not that
bad.  Nix doesn’t have grafts though, which is why this problem was less
visible there.

> Ultimately, I would prefer to see the scanning and grafting operations
> completely generalized, so that in general each package can specify how
> to scan and graft that particular package, making use of libraries in
> (guix build ...) to cover the usual cases.  In most cases, that code
> would be within build-systems.

That would be precise GC instead of conservative GC in a way, right?
So in essence we’d have, say, a scanner for ELF files (like ‘dh_shdep’
in Debian or whatever it’s called), a scanner for jars, and so on?
Still, how would we deal with strings embedded in the middle of
binaries, as in this case?  It seems to remain an open issue, no?

I’m interested in experiments in that direction.  I think that’s a
longer-term goal, though, and there are open questions: we have no idea
how well that would work in practice.

Thanks,
Ludo’.

bug#30875: Garbage collector may leave empty files

[Marius Bakke]

Hello,

Recently I've seen a couple of instances like these:

exporting path 
`/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz'
guix offload: error: build failed: hash of path 
`/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz' has changed 
from `4223b4813b2253e68ea2b824d0d0e284ff75714ad0faf5a33d65a78df6b65915' to 
`77ac62e2629d8e45f624589c0c8bf99e24b3a722349bf1e79bc186008534e246'!
cannot build derivation 
`/gnu/store/jbx6a4n1d55dpydr87d11m7xwapm3pcx-gnome-3.24.3.drv': 1 dependencies 
couldn't be built

Without the build hook, it manifests as:

starting phase `unpack'
tar: This does not look like a tar archive
xz: (stdin): File format not recognized
tar: Child returned status 1
tar: Error is not recoverable: exiting now

There was another instance on help-guix recently with a user having
empty .drv files in the store.

There is a bug lurking here somewhere, but I'm not sure where to start
looking.

$ find /gnu/store/ -maxdepth 1 -size 0 | grep -v '\.lock$' | wc -l
24


signature.asc
Description: PGP signature

bug#30841: Building Zend PHP (php): tests fail, and "Name or service not known"

[Adonay Felipe Nogueira]

> I also made my own "/etc/nsswitch.conf" file, which is attached. The

By the way I *did* notice the typo in "ndns" --- it should be "mdns"
---, and I corrected it, restarted nscd, restarted guix-daemon, and
tried again, with no good results.

I also tried "hosts: files dns", restarted nscd and guix-daemon,
respectively in this order, but also got no good results.

bug#30623: cdogs-sdl builds but cannot be run

[宋文武]

Nicolas Goaziou  writes:

> Hello,
>
> The package cdogs-sdl (release 0.6.6) seems to build fine, i.e., it is
> successfully added to the store. Yet, whenever I try to run it with
> "cdogs-sdl" command, it stops with a floating point exception.
>

Hello, I just updated the cdogs-sdl package to the latest git version,
which should fix this issue now.

Thanks for the report!

bug#30879: Commit bc499b113 broke guix on guile@2.0.14, improper field initialization

[Eric Bavier]

Hello Guix,

On the master branch (5d818b3557cc3b546d5bd0639359c14c7c0ab685), when
configured with guile@2.0.14, I get the following backtrace when
running `make`.

Backtrace:
In ice-9/boot-9.scm:
1739: 19 [#]
In unknown file:
   ?: 18 [primitive-load 
"/home/users/bavier/src/guix/./build-aux/compile-all.scm"]
In guix/build/compile.scm:
 158: 17 [compile-files "." "/home/users/bavier/src/guix" ...]
 107: 16 [load-files "." # # ...]
In ice-9/boot-9.scm:
2900: 15 [resolve-interface (gnu tests base) #:select ...]
2825: 14 [# # ...]
3101: 13 [try-module-autoload (gnu tests base) #f]
2412: 12 [save-module-excursion #]
3121: 11 [#]
In unknown file:
   ?: 10 [primitive-load-path "gnu/tests/base" ...]
In gnu/tests/base.scm:
 390: 9 [#]
  63: 8 [run-basic-test # # "basic" ...]
In ice-9/eval.scm:
 387: 7 [eval # #]
 387: 6 [eval # #]
 411: 5 [eval # #]
 387: 4 [eval # #]
In unknown file:
   ?: 3 [filter # (# # # #)]
In ice-9/eval.scm:
 411: 2 [eval # #]
 411: 1 [eval # #]
 387: 0 [eval # #]

ice-9/eval.scm:387:11: In procedure eval:
ice-9/eval.scm:387:11: In procedure mapped-device-target: Wrong type argument: 
#< device: "my-root" title: label mount-point: "/" type: "ext4" 
flags: () options: #f mount?: #t needed-for-boot?: #f check?: #t 
create-mount-point?: #f dependencies: () location: ((line . 209) (column . 24) 
(filename . "gnu/tests.scm"))>

(as an aside: maybe would could postpone compilation of test modules
until `make check`).

I git bisect'd this failure to commit
bc499b113a598c0e7863da9887a4133472985713, which added the
'initrd-modules' field to the (@ (gnu system) )
record.

The %simple-os from (gnu tests base) seems improperly initialized.  In
particular, the fields seem to be shifted:

scheme@(guile-user)> (@@ (gnu tests base) %simple-os)
$1 = #<
   kernel: #
   kernel-arguments: ()
   bootloader: #< bootloader: ...>
   initrd: #
   initrd-modules: ()
   firmware: "komputilo"
   host-name: #f
   hosts-file: ()
   mapped-devices: (#< device: "my-root" ...> #< 
...> ...)
   file-systems: ()
   swap-devices: (#< name: "alice" ...> ...)
   ...

Notice e.g. the "firmware" field has that value that should be in
"host-name", which has the value "hosts-file" should have, and
"mapped-devices" has the value "file-systems" should have, etc.

If you explicitely specify the new "initrd-modules" field this commit
added in (@ (gnu tests) %simple-os), then compilation proceeds as
expected.

-- 
Eric Bavier, Scientific Libraries, Cray Inc.

bug#30867: Troubles and solutions during install GuixSD

[Gábor Boskovits]

2018-03-19 23:41 GMT+01:00 Axel Koolhaas :

> Hello,
>
> I've recently installed GuixSD on my old laptop, but there were some
> problems where a Gentoo user even couldn't get past without IRC help.
>
>
> First i ran into old package problems, maybe mention on
> "https://www.gnu.org/software/guix/manual/html_node/Proceeding-with-the-
> Installation.html#Proceeding-with-the-Installation"
> 
> that one has to run "guix pull " to avoid these kind of errors:
>
> warning: collision encountered: 
> /gnu/store/hi7gadjf2gc3ma5bzabpkzjzcd89zz87-util-linux-2.30.1/share/man/man8/nologin.8.gz
>  
> /gnu/store/a8arlhcrf70113zw7b3qrwqbqfq2hgj6-shadow-4.5/share/man/man8/nologin.8.gz
> warning: arbitrarily choosing 
> /gnu/store/hi7gadjf2gc3ma5bzabpkzjzcd89zz87-util-linux-2.30.1/share/man/man8/nologin.8.gz
> Backtrace:
>5 (primitive-load "/gnu/store/nai4dbb66sv8b1c7s8dh10kf13h…")
> In ./guix/build/profiles.scm:
> 137:2  4 (build-profile "/gnu/store/xwjp5021qhc5y75spvj2wgvhb9y…" …)
> In unknown file:
>3 (hash-for-each # …)
>2 (hash-for-each # …)
>1 (hash-for-each # …)
>0 (scm-error misc-error #f "~A ~S" ("union-build: col…" …) …)
>
> ERROR: In procedure scm-error:
> ERROR: union-build: collision between file and directories ((files 
> ("/gnu/store/2m4kj0py6dcsm33r4h0wgmm7cqc9fhqi-nano-2.9.1/share/locale/es")) 
> (dirs 
> ("/gnu/store/hi7gadjf2gc3ma5bzabpkzjzcd89zz87-util-linux-2.30.1/share/locale/es"
>  "/gnu/store/a8arlhcrf70113zw7b3qrwqbqfq2hgj6-shadow-4.5/share/locale/es" 
> "/gnu/store/w3h1kpg13s8d12rddqkjv29xzidxwxyj-man-db-2.7.6.1/share/locale/es" 
> "/gnu/store/w7mj8n7lzmwihs58hipfwcih82j31n60-info-reader-6.3/share/locale/es" 
> "/gnu/store/9gww54njmvi0isyhlxabbzz80bx751zs-sudo-1.8.21p2/share/locale/es" 
> "/gnu/store/pwl7clzd9z87vrgbk5lqp49glddn432v-e2fsprogs-1.43.6/share/locale/es"
>  "/gnu/store/6b6mps3s9nmavlixkkifmp93fkzv13wd-kbd-2.0.4/share/locale/es" 
> "/gnu/store/ars9lm9jk9hgdifg0gqvf1jrvz5mdg1j-bash-4.4.12/share/locale/es" 
> "/gnu/store/jq01kjk404gwyjz0xnr098zh20b5i0n3-coreutils-8.27/share/locale/es" 
> "/gnu/store/44hxccx5q9dwai01d3ngvjs23ng4ynz3-findutils-4.6.0/share/locale/es" 
> "/gnu/store/ccsanpky6z3f4w5gd75x3i0xgv9yalw7-grep-3.0/share/locale/es" 
> "/gnu/store/s5n8rml09apz4ld8g9j3k9fjh8mkpb83-sed-4.4/share/locale/es" 
> "/gnu/store/nlk8wizl96yf9xrzgis41vqwdfzyhj8s-diffutils-3.6/share/locale/es" 
> "/gnu/store/50kx5jp0ccr2vk0q5q6ini1r1ajyqxqk-gawk-4.1.4/share/locale/es" 
> "/gnu/store/30zazbgh0lx2pav0ikx5hglgay28264j-tar-1.29/share/locale/es")))
> builder for `/gnu/store/1sm0amip5hdqk2wmjcwh207y022wgc7s-profile.drv' failed 
> with exit code 1
> cannot build derivation 
> `/gnu/store/zd61d3qda8ij0pkxc5kiyf7rxk0xyp26-system.drv': 1 dependencies 
> couldn't be built
> guix system: error: build failed: build of 
> `/gnu/store/zd61d3qda8ij0pkxc5kiyf7rxk0xyp26-system.drv' failed
>
>
> Secondly, in the manual now, it appears as if to mount the EFI System
> Partition /mnt/boot/efi, whilst one needs to mount ESP at /boot/efi,
> resulting in this kind of error:
>
> /gnu/store/nwg3rkmb2is1j6zfin0v58lgw2b57kwh-system
> /gnu/store/r8qpnbydygbwdfsjcla4ms5rjikbmkjv-grub.cfg
> /gnu/store/nkn9f21jjzcj0a3mr4sqc0qj549g8vfl-grub-efi-2.02
> /gnu/store/gjb81f7h063l83rlfb9ih1vrsg9k0z4l-bootloader-installer
>
> initializing operating system under '/mnt'...
> copying to '/mnt'...
> populating '/mnt'...
> Installing for x86_64-efi platform.
> /gnu/store/nkn9f21jjzcj0a3mr4sqc0qj549g8vfl-grub-efi-2.02/sbin/grub-install: 
> error: failed to get canonical path of `/boot/efi'.
> guix system: error: failed to install bootloader 
> /gnu/store/gjb81f7h063l83rlfb9ih1vrsg9k0z4l-bootloader-installer
>
>
> Thirdly, whilst this isn't a bug but more noob user-friendliness, maybe
> add a link or an example of a working EFI config to manual?
> I now had to search before finding e.g.
>
> (file-systems (cons* (file-system
>  (device "my-root")
>  (title 'label)
>  (mount-point "/")
>  (type "ext4"))
>(file-system
>  (device (uuid "1234-ABCD" 'fat))
>  (title 'uuid)
>  (mount-point "/boot/efi")
>  (type "vfat"))
>%base-file-systems))
>
> Also, whilst I know this but not all other people, one needs to mount
> ESP's with "mount -t vfat /dev/sda* /boot/efi", maybe also mention this in
> the guide?
>
> Kind regards,
> Shoaloak
>
Thanks very much for your findings, we currently have a few EFI
installation documentation related bugs, this is being worked on.
We also intend to add system tests, so that we can check, if the
documentation is in sync with the code.
I will check if there is a bug this should be merged to.

bug#30395: bug#30820: Chunked store references in compiled code break grafting (again)

[Ludovic Courtès]

Hello,

l...@gnu.org (Ludovic Courtès) skribis:

> So the real issue is this:
>
>> The second issue is that the patch only ever worked with literal
>> strings.  It does not “see” strings in constant arrays like the ‘str’
>> array in the example above.

Good news!  Commit e288572710250bcd2aa0f69ce88154d98ac69b29 adjusts
‘gcc-strmov-store-file-names.patch’ in ‘core-updates’ to correctly deal
with this case:

--8<---cut here---start->8---
$ cat strmov.c 
#define _GNU_SOURCE
#include 
static const char str[] =
  "This is a /gnu/store/ee string in a global 
variable.";

extern char *p, *q;

#ifndef MEMCPY
# define MEMCPY memcpy
#endif

void foo (char *x, char *y)
{
  MEMCPY (x, str, sizeof str);
  MEMCPY (y, "this is a literal /gnu/store/ee 
string", 35);
}
$ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) gcc-final)'
/gnu/store/wzdyqkdslk1s6f0vi9qw1xha8cniijzs-gcc-5.5.0-lib
/gnu/store/46ww5s9zvsw04id438c4drpnwd9m6vl8-gcc-5.5.0
$ /gnu/store/46ww5s9zvsw04id438c4drpnwd9m6vl8-gcc-5.5.0/bin/gcc -O2 -c strmov.c
$ objdump -S strmov.o |grep movabs
$ NIX_STORE=/foo /gnu/store/46ww5s9zvsw04id438c4drpnwd9m6vl8-gcc-5.5.0/bin/gcc 
-O2 -c strmov.c
$ objdump -S strmov.o |grep movabs
   0:   48 b8 54 68 69 73 20movabs $0x2073692073696854,%rax
   a:   48 ba 74 6f 72 65 2fmovabs $0x6565652f65726f74,%rdx
  1e:   48 b8 61 20 2f 67 6emovabs $0x732f756e672f2061,%rax
  30:   48 b8 65 65 65 65 65movabs $0x6565656565656565,%rax
  4a:   48 b8 65 65 65 65 65movabs $0x2065656565656565,%rax
  58:   48 b8 73 74 72 69 6emovabs $0x6920676e69727473,%rax
  66:   48 b8 6e 20 61 20 67movabs $0x626f6c672061206e,%rax
  74:   48 b8 61 6c 20 76 61movabs $0x6169726176206c61,%rax
  82:   48 b8 74 68 69 73 20movabs $0x2073692073696874,%rax
  93:   48 b8 61 20 6c 69 74movabs $0x61726574696c2061,%rax
  a5:   48 b8 6c 20 2f 67 6emovabs $0x732f756e672f206c,%rax
--8<---cut here---end--->8---

I built everything about to ‘gcc-final’ in ‘core-updates’.  I checked
manually that none of the /gnu/store references in libc-2.26.so were
chunked.

For the record, what the patch initially did was to skip code that would
otherwise emit a “block move” when expanding __builtin_memcpy & co.
This patch additionally skips similar code that would replace
__builtin_memcpy calls with memory moves early on, in
‘gimple_fold_builtin_memory_op’, before ‘expand_builtin’ is called.

In the example above, this transformation would lead to the code below
(as seen with ‘-fdump-tree-all’ in the ‘gimple’ phase output):

--8<---cut here---start->8---
foo (char * x, char * y)
{
  MEM[(char * {ref-all})x] = MEM[(char * {ref-all})&str];
  memcpy (y, "this is a literal /gnu/store/ee 
string", 35);
}
--8<---cut here---end--->8---

With the patch we get:

--8<---cut here---start->8---
foo (char * x, char * y)
{
  memcpy (x, &str, 85);
  memcpy (y, "this is a literal /gnu/store/ee 
string", 35);
}
--8<---cut here---end--->8---

Ludo’.

bug#30875: Garbage collector may leave empty files

[Ludovic Courtès]

Hello,

Marius Bakke  skribis:

> Recently I've seen a couple of instances like these:
>
> exporting path 
> `/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz'
> guix offload: error: build failed: hash of path 
> `/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz' has 
> changed from 
> `4223b4813b2253e68ea2b824d0d0e284ff75714ad0faf5a33d65a78df6b65915' to 
> `77ac62e2629d8e45f624589c0c8bf99e24b3a722349bf1e79bc186008534e246'!
> cannot build derivation 
> `/gnu/store/jbx6a4n1d55dpydr87d11m7xwapm3pcx-gnome-3.24.3.drv': 1 
> dependencies couldn't be built
>
> Without the build hook, it manifests as:
>
> starting phase `unpack'
> tar: This does not look like a tar archive
> xz: (stdin): File format not recognized
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> There was another instance on help-guix recently with a user having
> empty .drv files in the store.
>
> There is a bug lurking here somewhere, but I'm not sure where to start
> looking.
>
> $ find /gnu/store/ -maxdepth 1 -size 0 | grep -v '\.lock$' | wc -l
> 24

Are we sure this is a Guix issue and not a disk or file system
corruption issue?  The relevant code in guix-daemon hasn’t changed in
ages.

What file system are you using?  Btrfs?  :-)

Thanks,
Ludo’.

bug#30879: Commit bc499b113 broke guix on guile@2.0.14, improper field initialization

[Ludovic Courtès]

Hello Eric,

Eric Bavier  skribis:

> scheme@(guile-user)> (@@ (gnu tests base) %simple-os)
> $1 = #<
>kernel: #
>kernel-arguments: ()
>bootloader: #< bootloader: ...>
>initrd: #
>initrd-modules: ()
>firmware: "komputilo"
>host-name: #f
>hosts-file: ()
>mapped-devices: (#< device: "my-root" ...> 
> #< ...> ...)
>file-systems: ()
>swap-devices: (#< name: "alice" ...> ...)
>...
>
> Notice e.g. the "firmware" field has that value that should be in
> "host-name", which has the value "hosts-file" should have, and
> "mapped-devices" has the value "file-systems" should have, etc.
>
> If you explicitely specify the new "initrd-modules" field this commit
> added in (@ (gnu tests) %simple-os), then compilation proceeds as
> expected.

That sounds a lot like regular ABI breakage: a new 
field was added but gnu/tests/base.go wasn’t rebuilt, and thus was
expecting the previous struct layout.

Does “rm gnu/tests/base.go && make” suffice to fix this issue?

Thanks,
Ludo’.

bug#30890: Hash mismatch for /gnu/store/bvbs19jg8497ca73i82xmrjspd83lvs2-usb.ids

[Vivien Kraus]

Hello,

I tried to install gnome with guix, but it fails when building this:

Starting download of /gnu/store/bvbs19jg8497ca73i82xmrjspd83lvs2-
usb.ids
>From http://linux-usb.cvs.sourceforge.net/viewvc/linux-usb/htdocs/usb.i
ds?revision=1.551...
 ids  4KiB0B/s 00:00
[ids  4KiB274KiB/s 00:00
[### ids  4KiB259KiB/s 00:00
[##] 100.0%
sha256 hash mismatch for output path
`/gnu/store/bvbs19jg8497ca73i82xmrjspd83lvs2-usb.ids'
  expected: 17rg5i0wbyk289gr8v4kgvnc9q5bidz7ldcvv9x58l083wn16hq3
  actual:   10mcg24vdvbbm9lk2scrfs7ff7n2a0dkl2qlkzzllhn53yyrvkc6
cannot build derivation `/gnu/store/y7ahn1vd09phqxpz5shgwdggl41rd70a-
libosinfo-1.0.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/9h84d02l7d8n9l0n0mnqj2294nzmk4wk-
tracker-1.12.3.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/ckf5yv041kdm0barjv0ksg1hq0hvizy9-
nautilus-3.26.2.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/0c0mkrgwrg4v70gn2rrcqrgk6b7l13g3-
gnome-3.24.3.drv': 1 dependencies couldn't be built
guix package: error: build failed: build of
`/gnu/store/0c0mkrgwrg4v70gn2rrcqrgk6b7l13g3-gnome-3.24.3.drv' failed

What should I do about this?  Should I trust this?  If so, how should I
proceed?

Regards,

Vivien

bug#30890: Hash mismatch for /gnu/store/bvbs19jg8497ca73i82xmrjspd83lvs2-usb.ids

[Danny Milosavljevic]

Hi,

apparently linux-usb sourceforge switched over to SVN - so what you are getting
there is an error page.

Possible fix would be:

diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm
index 55a92eca0..be8a8bb86 100644
--- a/gnu/packages/virtualization.scm
+++ b/gnu/packages/virtualization.scm
@@ -280,7 +280,7 @@ server and embedded PowerPC, and S390 guests.")
("usb.ids"
 ,(origin
(method url-fetch)
-   (uri 
"http://linux-usb.cvs.sourceforge.net/viewvc/linux-usb/htdocs/usb.ids?revision=1.551";)
+   (uri 
"https://svn.code.sf.net/p/linux-usb/repo/trunk/htdocs/usb.ids?r=1551";)
(file-name "usb.ids")
(sha256
 (base32



pgpOiehKlaWry.pgp
Description: OpenPGP digital signature

bug#30875: Garbage collector may leave empty files

[Mark H Weaver]

l...@gnu.org (Ludovic Courtès) writes:

> Marius Bakke  skribis:
>
>> Recently I've seen a couple of instances like these:
>>
>> exporting path 
>> `/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz'
>> guix offload: error: build failed: hash of path
>> `/gnu/store/mi1rbw8fdsbi4bc4pndbb6smq39722vm-baobab-3.26.1.tar.xz'
>> has changed from
>> `4223b4813b2253e68ea2b824d0d0e284ff75714ad0faf5a33d65a78df6b65915'
>> to
>> `77ac62e2629d8e45f624589c0c8bf99e24b3a722349bf1e79bc186008534e246'!
>> cannot build derivation 
>> `/gnu/store/jbx6a4n1d55dpydr87d11m7xwapm3pcx-gnome-3.24.3.drv': 1 
>> dependencies couldn't be built
>>
>> Without the build hook, it manifests as:
>>
>> starting phase `unpack'
>> tar: This does not look like a tar archive
>> xz: (stdin): File format not recognized
>> tar: Child returned status 1
>> tar: Error is not recoverable: exiting now
>>
>> There was another instance on help-guix recently with a user having
>> empty .drv files in the store.
>>
>> There is a bug lurking here somewhere, but I'm not sure where to start
>> looking.
>>
>> $ find /gnu/store/ -maxdepth 1 -size 0 | grep -v '\.lock$' | wc -l
>> 24
>
> Are we sure this is a Guix issue and not a disk or file system
> corruption issue?  The relevant code in guix-daemon hasn’t changed in
> ages.
>
> What file system are you using?  Btrfs?  :-)

The ext[234] filesystems are well known for leaving zero-length files
around after a crash.  So far, I've never seen Btrfs do that, and I
wouldn't expect it to based on its design.  That's partly why I switched
to it.

  Mark

bug#30820: Chunked store references in compiled code break grafting (again)

[Mark H Weaver]

Hi Ludovic,

l...@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver  skribis:
>
>> We would also need to find a solution to the problem described in the
>> thread "broken references in jar manifests" on guix-devel started by
>> Ricardo, which still has not found a satifactory solution.
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2018-03/msg6.html

Okay, do you have a proposed fix for the issue of jar manifests?

There's a specification for that file format which mandates that "No
line may be longer than 72 bytes (not characters), in its UTF8-encoded
form.  If a value would make the initial line longer than this, it
should be continued on extra lines (each starting with a single SPACE)."

>> My opinion is that I consider Guix's current expectations for how
>> software must store its data on disk to be far too onerous, in cases
>> where that data might include a store reference.  I don't see sufficient
>> justification for imposing such an onerous requirement on the software
>> in Guix.
>
> In practice Guix and Nix have been living fine under these constraints,
> and with almost no modifications to upstream software, so it’s not that
> bad.  Nix doesn’t have grafts though, which is why this problem was less
> visible there.
>
>> Ultimately, I would prefer to see the scanning and grafting operations
>> completely generalized, so that in general each package can specify how
>> to scan and graft that particular package, making use of libraries in
>> (guix build ...) to cover the usual cases.  In most cases, that code
>> would be within build-systems.
>
> That would be precise GC instead of conservative GC in a way, right?
> So in essence we’d have, say, a scanner for ELF files (like ‘dh_shdep’
> in Debian or whatever it’s called), a scanner for jars, and so on?

No, I wasn't thinking along those lines.  While I'd very much prefer
precise GC, it seems wholly infeasible for us to write precise scanners
and grafters for every file format of every package in Guix.

My thought was that supporting scanning and grafting of 8-byte-or-longer
substrings of hashes would cover both GCC's inlined strings and jar
manifests, the two issues that we currently know about, and that it
would be nice if we could add further methods in the future.  For
example, some software might store its data in UTF-16, or compressed.

> Still, how would we deal with strings embedded in the middle of
> binaries, as in this case?  It seems to remain an open issue, no?

I believe that I addressed that case in my original proposal, no?

> I’m interested in experiments in that direction.  I think that’s a
> longer-term goal, though, and there are open questions: we have no idea
> how well that would work in practice.

Thanks for discussing it.  I'm willing to drop it and go with your
decision for now, but the "jar manifest" issue still needs a solution.

 Regards,
   Mark

bug#30820: Chunked store references in compiled code break grafting (again)

[Mark H Weaver]

I just realized that my proposal is unworkable.  If we allow strings
containing store paths to be split into pieces, then some of those
pieces may contain as little as one character of the hash.  For example,
the grafter might find "/store/c", which is likely not enough to
determine which of the transitive inputs is being referenced, and
therefore the grafter cannot decide what to write in place of the "c".

Sorry for the noise.

   Mark

bug#30395: bug#30820: Chunked store references in compiled code break grafting (again)

[Ricardo Wurmus]

Hi Ludo,

> Good news!  Commit e288572710250bcd2aa0f69ce88154d98ac69b29 adjusts
> ‘gcc-strmov-store-file-names.patch’ in ‘core-updates’ to correctly deal
> with this case:
[…]
> I built everything about to ‘gcc-final’ in ‘core-updates’.  I checked
> manually that none of the /gnu/store references in libc-2.26.so were
> chunked.

Wow, thank you so much for fixing this!

Is this an option that we can suggest to the GCC developers to
officially support?

-- 
Ricardo