Re: avoid wrapper scripts when possible

[Ricardo Wurmus]

Christopher Baines  writes:

> However, I think that the file wrapping approach has advantages for
> visibility. Maybe it could be tweaked to keep ensure the wrapper script
> has the same name as the script its wrapping, e.g. when wrapping foo,
> replace foo with a bash script, and move the real script
> to .wrapped-bin/foo.

I agree that having a separate wrapper is certainly nicer than patching
upstream scripts automatically.  The idea of moving original scripts out
of the way sounds intriguing, but I fear that we would have a similar
set of problems.

In the case of conda’s “activate” script there would be an obvious
problem: the script tries to determine the directory that contains it.
If that were a different directory than the one the developers intended,
it would fail in unexpected ways.

I wonder if there’s a way to have a script header that starts out as a
shell script and then calls itself again but skipping over the header,
thus calling the original script.

Take this Python script as an example:

--8<---cut here---start->8---
#!/home/rekado/.guix-profile/bin/python
print("hello from python")
--8<---cut here---end--->8---

We then prepend this header:

--8<---cut here---start->8---
#!/run/current-system/profile/bin/bash
echo "hello from bash"
exec /home/rekado/.guix-profile/bin/python 
<(/run/current-system/profile/bin/tail -n +4 "$0")
--8<---cut here---end--->8---

When we execute the resulting thing it will be a shell script, and do
shell things printing “hello from bash”.  Then it scrubs itself off the
script and removes the original script’s Python shebang using tail (with
“-n +4” where 4 is the number of lines of the shell script header + 1),
and replaces itself with Python calling the original script.

This seems to work just fine and off the top of my head I can’t think of
a situation where this would fail — unless the wrapped script performed
the same kind of trick of reading its contents through $0.  (How likely
is that?)

What do you think?  Is this too weird to consider?  Weirder than
renaming binaries and replacing them with scripts?

One side effect would be that “file” would be rather confused about
these scripts, but maybe that’s acceptable.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: avoid wrapper scripts when possible

[Hartmut Goebel]

Hi,

answering on a mail which Ricardo CCed to me and which did not make it
to the list yet. Thus I full-quote.

Am 04.11.2017 um 23:30 schrieb Ricardo Wurmus:
> How about this:
>
> --8<---cut here---start->8---
> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile
> #!# (let (($0 (car (command-line (execl 
> "/home/rekado/.guix-profile/bin/python3" $0 $0))
> #!/home/rekado/.guix-profile/bin/python3
> import sys; print("hello from python: "+sys.argv[0])
> --8<---cut here---end--->8---
>
> The first two lines are Guile code, but they are also line comments in
> shell, Perl, Python, Ruby, and R.  The Guile code in this example calls
> the script again as a Python script.  Before doing that it can set
> environment variables, like so:
>
> --8<---cut here---start->8---
> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile
> #!#
> #\- (setenv "PYTHONPATH" (string-append "/gnu/store/foo:/gnu/store/bar:" 
> (getenv "PYTHONPATH")))
> #\- (let (($0 (car (command-line (execl 
> "/home/rekado/.guix-profile/bin/python3" $0 $0))
> #!/home/rekado/.guix-profile/bin/python3
> import sys; print("hello from python: "+sys.argv[0])
> print(sys.path)
> --8<---cut here---end--->8---
>
> Did I overlook something?  Or could we use this hack to remove separate
> wrapper scripts for Perl, Python, Ruby, and R?


I testes this and (beside some necessary adjustments) this works. Indeed
this a very elegant hack :-). My respect! Compared to my idea this is
more elegant since it does not require a separate file. 

Some fine-tuning needs to be done, though:

1) The setenv-line should only append the existing env-var, if this was
set. (At least in Python, an empty PYTHONPATH part is interpreted as
"current directory", thus adding an empty part would change the desired
behavior.)

2) Arguments need to be passed to the script, too.

Since I was eager to learn how this can be done, here is my draft fixing
these two issues:

--8<---cut here---start->8---

#!/bin/guile --no-auto-compile
#!#; Guix wrapping header
#\- (define (append-env name path) (let ((var (getenv name))) (setenv
name (if var (string-append path ":" var) path
#\- (append-env "PYTHONPATH" "/gnu/store/foo:/gnu/store/bar")
#\- (apply execl "/bin/python3" (car (command-line)) (command-line))
#!/home/rekado/.guix-profile/bin/python3

print("This is the real program")
import sys, os
print("sys.path[:4]", sys.path[:4])
print("argumentes:", sys.argv)
print("ppid", os.getppid())

--8<---cut here---end--->8---

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: avoid wrapper scripts when possible

[Hartmut Goebel]

Hi,

Ricado proposed a new and better solution already. Nevertheless I want
to comment on this proposal, sint it includes a pitfall and would not work:

Am 03.11.2017 um 22:17 schrieb Ricardo Wurmus:
> exec /home/rekado/.guix-profile/bin/python 
> <(/run/current-system/profile/bin/tail -n +4 "$0")
[...]
> This seems to work just fine and off the top of my head I can’t think of
> a situation where this would fail — unless the wrapped script performed
> the same kind of trick of reading its contents through $0.  (How likely
> is that?)

Indeed for Python programs it is quite common to revert to the filename
using the variable "__file__". And if feeding in the script from stdin
would set __file__ to "". Using __file__ is more often used in
package modules than in scripts (which could use $0 (sys.argv[0]).
Anyway this is a pitfall we should avoid.

Please not that Ricardo's new solution, based on guile and execl does
not suffer from this problem (I verified).

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: NFS mounts

[Konrad Hinsen]

Hi Ludo,

> By default, file systems are automatically mounted at boot time.  To
> avoid that, you must add:
>
>   (mount? #f)

That works well indeed. But I actually want that NFS filesystem mounted
at boot time, ideally (not strictly required though).

Of course, the real problem is that I cannot mount it at all because
rpc.statd is missing.

Konrad.

avoid downloading bootstrap binaries?

[Ricardo Wurmus]

Hi Guix,

we have some Makefile rules that download some bootstrap binaries for
us.  Why is this necessary?  Could we ship these bootstrap binaries
instead?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Building Docker images of GuixSD

[Ludovic Courtès]

Hi!

Chris Marusich  skribis:

> l...@gnu.org (Ludovic Courtès) writes:

[...]

>> Or we mount the host store over 9p and exec a dynamically-linked Guile
>> from there.
>
> I understand hat by "9p" you mean to share the file system with the VM,
> but I don't quite understand what this option entails.  If the initrd
> doesn't use the right Guile, why will sharing the store via 9p work?  Do
> you mean to exec directly somehow, instead of using an initrd...?

9p support is in the kernel, so we can always mount a 9p file system,
provided the relevant 9p kernel modules are loaded.

HTH!

Ludo’.

Re: GDM status (again)

[Ludovic Courtès]

Hi!

Timothy Sample  skribis:

> Andy Wingo  writes:
>
>>> Do we really need a compile-time dependency (GDM in master doesn’t
>>> depend on gnome-shell, after all), or can we just have a hard-coded
>>> /run/current-system/bin/gnome-shell somewhere?  Granted, that’s not very
>>> elegant, but it might be good enough?
>>>
>>> Also, one can use GDM without using GNOME, so it would be best if GDM
>>> didn’t depend on GNOME.
>>
>> This might be fine (and is what I was trying to do).  However this would
>> mean that the GDM service should ensure that gnome-shell is there, which
>> in practice is like having a dependency on GNOME or at least on
>> gnome-shell.  My understanding is that GDM uses gnome-shell as the login
>> greeter process, so you really can't use GDM without GNOME -- of course
>> you can log in to some non-GNOME session but GDM itself will use
>> gnome-shell.  Given that's the case, might as well depend on gnome-shell
>> explicitly.
>
> That’s my understanding too. GDM needs gnome-shell to run the greeter
> interface.

Oh I didn’t know that.  I agree then, it’s best to explicitly depend on it.

Thank you!

Ludo’.

Re: Status update on reproducible builds in Guix

[Ludovic Courtès]

Jan Nieuwenhuizen  skribis:

> Ludovic Courtès writes:
>
>> Here’s an update on reproducibility in Guix:
>>
>>   
>> https://www.gnu.org/software/guix/news/reproducible-builds-a-status-update.html
>
> At least 78% to possibly 91% reproduciblility of packages is not bad.
>
> Is there a (small) core that is already 100% reprocucible, like the
> installation binaries/USB installer, bare-bones.tmpl or even
> lightweight-desktop.tmpl?

Good question!  I think as soon as you have Python .pyc files in the
dependency graph (reference graph), there are non-reproducible bits.  We
certainly have Python stuff in the base system reference graph, so
that’s one thing we should fix most urgently I guess.

Ludo’.

Re: why is linux-libre-headers behind linux-libre?

[Ludovic Courtès]

Hello,

Dave Love  skribis:

> Efraim Flashner  writes:
>
>> On Tue, Oct 31, 2017 at 02:00:35PM +0100, Vincent Legoll wrote:
>>> Hello,
>>> 
>>> On Tue, Oct 31, 2017 at 1:35 PM, Dave Love  wrote:
>>> > Why is linux-libre-headers a long way behind linux-libre (currently at
>>> > version 4.4.47, compared with 4.13.10 for linux-libre)?
>>> 
>>> I suspect this is due to massive rebuilding that would occur when
>>> updating linux-libre-headers

That and also because glibc targets (can target) older kernels, which is
something we rely on.

>> This is typically updated in the core-updates branch, but it hasn't been
>> updated yet. Based on the LTS versions, we should upgrade it to the 4.9
>> branch.
>
> Thanks for the explanations.  I checked that 4.9 would support the
> Omnipath library, at least.

The Omnipath library relies on Linux (not libc) headers, and a specific
version thereof?

I suppose we could also introduce a more recent version of
‘linux-libre-headers’ specifically for this purpose, with the
understanding that the resulting binaries rely on a specific kernel
version.

Thanks,
Ludo’.

Re: Tiny Guix (and containers)

[Ludovic Courtès]

Dave Love  skribis:

> Pjotr Prins  writes:
>
>> But, really, I think when talking embedded systems and containers we
>> all want tiny. Even HPC can benefit. Tiny containers may be an
>> attractive proposition.
>
> Yes, containers aside, dependencies in Guix are one of the reasons I'm
> currently unconvinced by its trades-off for HPC use; the closure a
> single package can be comparable with the whole compute node image I
> used to maintain with rpm.  Even then, you don't generally have
> debugging info available.
>
> I suppose the general point is that Guix seems to have rejected
> experience from other distributions, and it's not clear to me why.  (I
> don't mean it should necessarily follow them, of course.)  Is there any
> summary of the rationale for different decisions like not splitting
> packages into development and runtime and other components, packaging
> static libraries, and discarding debugging information, for instance?

The main reason, as you have noticed, is that multiple outputs don’t
always work out-of-the-box: the GC scanner does not handle circular
dependencies among outputs of a given derivation, which makes things
more difficult.

Another reason is that splitting is just part of the story.  Often, it’s
not moving out 10 KB of header files that really helps; rather, it’s
stripping the dependency graph.

Some people also argued that having everything in one package made
things easier for them as users, though that one is more questionable to
me.

Ludo’.

Re: Tiny Guix (and containers)

[Ludovic Courtès]

Heya,

Dave Love  skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> Hi,
>>
>> Hartmut Goebel  skribis:
>
>>> I'm in favor of (automatically?) splitting of "development" packages,
>>> including the headers and the static libs (much like the "-devel" or
>>> "-dev" packages in other distributions. One does not need them on a
>>> production system and they are just wasting space. When Guix needs to
>>> build a package, it automatically installs the ":devel" output of all
>>> it's inputs.
>
> I've been arguing for making sub-packages similar to other
> distributions, but I'd expect to specify something like :devel as the
> input to builds.  I'm not sure that it would even work generally to
> infer it (e.g. when cross-compiling).

OK.

>> We could do that (the Nixpkgs folks did exactly that a while back), but
>> it won’t help that much.
>
> It looks to me as if it would often help significantly, e.g. when a
> pkg-config file, or something else sucks in a load of stuff that's
> irrelevant for running the package.  (Separating :lib and needing that
> for building means you need to know something about the packaging rather
> than just using "devel", say.)

Right, good point.

The nice thing with “lib” and “doc” is that it has a direct mapping to
the GNU directory classification (libdir, docdir, etc.)

Now, we could depart from it and go with “devel”, for the reasons you
give.  Let’s experiment and see how it goes!

>> What does help is running ‘guix size’, looking
>> at specific packages that are problematic, then finding a solution for
>> these packages—and often enough the solution is non-trivial.
>
> I think it often will reflect the lack of separation of development
> files, documentation, etc., and inclusion of static libraries, for
> instance.  Boost is a case in point.  There's also the issue of multiple
> copies/versions of packages in the closure, which seems problematic for
> more than just size reasons.

Boost is also a special case, being a mostly-header library.  :-)  But
yeah, I get your point.

>> But yeah, I think we should track packages that are big or have a
>> surprisingly build closure, and try to fix that incrementally!
>
> Shouldn't that be done as a matter of course?  I don't remember if it's
> part of Fedora or Debian packaging guidelines but packagers should check
> dependencies for sanity when packaging, and there's some detection of
> unnecessary linkage.  Guix' Qt dependencies are a striking example which
> looks hard to resolve.  Generally I get surprised at what typically gets
> built -- at great length! -- on updates or installation.

No argument here!

The guidelines at

also mention the closure size.

We can clearly do better though, and what you’ve been doing with
Open MPI et al. is a step in the right direction IMO.

I would very much welcome experiments introducing “devel” outputs.
Perhaps as a first step we could make it opt-in.  We would add support
for that in gnu-build-system.scm, and then modify key packages to use
“devel” instead of “lib”, for example.  Incremental changes like this
can be tractable.

Thoughts?

Ludo’.

Re: [bootstrappable] diverse double compilation: using $ORIGIN?

[Ludovic Courtès]

Jan Nieuwenhuizen  skribis:

> From c91609e847066c384826d726033146e08d8185ed Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen 
> Date: Thu, 2 Nov 2017 06:52:46 +0100
> Subject: [PATCH] gnu: Add clang-gcc, gcc-ddc.  WIP
>
> Usage: guix build gcc-dcc
>
> Building gcc-dcc tests the diverse double compilation property
> of the gcc that Guix is using.
>
> gcc-dcc is a meta-package that depends on gcc-5.4.0 and on
> clang-gcc-5.4.0 (the same GCC built with clang).  The builder
> checks if both gcc's are bit-for-bit identical and fails if
> they differ.
>
> * gnu/packages/bootstrappable.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.

Awesome!  Does it build fine out-of-the-box?  I didn’t expect it to be
“this easy.”  :-)

Ludo’.

Re: [bootstrappable] diverse double compilation: using $ORIGIN?

[Ludovic Courtès]

Hello!

Ricardo Wurmus  skribis:

> When building packages, Guix embeds the absolute file name of the output
> directory in the resulting binary.  That’s usually fine and what we
> want.
>
> Now let’s assume that we’d like to build the GCC sources with diverse
> double compilation: we’d have a “gcc” package that’s built with
> “gcc-core”; we’d also have a “gcc-from-clang” package that uses Clang as
> its input.  Since the GCC build procedure is performed at least two
> times (once with the bootstrap compiler, and then again with the GCC
> variant this produces), the resulting GCC binaries should be identical.
>
> Except that they are not.  One of the reasons is that the binaries that
> Guix produces embed the target output directories.  This means that the
> two compiler binaries that result from diverse double compilation will
> *always* differ in at least the embedded paths, such as paths to itself
> (e.g. to binaries in the libexec directory) and paths to.
>
> I wonder if we can use $ORIGIN in the compiler binaries, so that we can
> avoid (most) references to the output directories, thereby making the
> equality check between the diversely built compiler binaries simpler.

Assuming we can add a phase to replace absolute file names in RUNPATH &
co. with $ORIGIN references where applicable (should be doable), we
still have a problem with multiple-output derivations: we’d still have
the hash of the other outputs.

I’m not sure how to avoid that.

When the goal is just to compare things, we could build static
binaries.  Or we could have a comparison tool that compares modulo file
names or something…

Thoughts?

Ludo’.

Re: Disable gnulib's test-lock test in packages?

[Ludovic Courtès]

Hi!

Eric Bavier  skribis:

> I ran the gnulib lock tests with the be95b17ae commit and the tests passed, 
> then with the commit directly before and they failed; so I'm fairly confident.

Great, thanks for testing.

> Attached is an updated patch, which remove the *-gnulib-multi-core patches 
> and disables the test on two more packages (augeas and gsasl).  I was able to 
> build most of the packages on gnulib's "users" list 
> (https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=users.txt;h=4ea176f2ef622b5c12843ba3d5aa35217c58502b;hb=HEAD)
>  except for octave, because texlive-bin's luajit tests fail on this system.

OK.

> From 8e4988cf0da5bb170184a41822ff316e2e7f9c29 Mon Sep 17 00:00:00 2001
> From: Eric Bavier 
> Date: Mon, 30 Oct 2017 11:23:46 -0500
> Subject: [PATCH] gnu: Disable gnulib's test-lock test in packages.
>
> * gnu/packages/base.scm (findutils)[source]: Disable test-lock.
> * gnu/packages/gettext.scm (gettext-minimal)[source]: Ditto.
> * gnu/packages/libidn.scm (libidn)[source]: Ditto.
> * gnu/packages/libunistring.scm (libunistring)[source]: Ditto.
> * gnu/packages/augeas.scm (augeas)[source]: Ditto.
> * gnu/packages/gsasl.scm (gsasl)[source]: Ditto.
> * gnu/packages/patches/findutils-gnulib-multi-core.patch,
> gnu/packages/patches/gettext-gnulib-multi-core.patch,
> gnu/packages/patches/gettext-multi-core.patch,
> gnu/packages/patches/libunistring-gnulib-multi-core.patch: Delete patches.
> * gnu/local.mk (DIST_PATCH_DATA): Remove them.

OK for ‘core-updates’!

Thank you,
Ludo’.

Re: [bootstrappable] diverse double compilation: using $ORIGIN?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Usage: guix build gcc-dcc
>>
>> Building gcc-dcc tests the diverse double compilation property
>> of the gcc that Guix is using.
>>
>> * gnu/packages/bootstrappable.scm: New file.
>> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
>
> Awesome!  Does it build fine out-of-the-box?  I didn’t expect it to be
> “this easy.”  :-)

I got very positive feedback from a slightly guix-sceptic person saying
something like: you lisp guys can do quite a lot with very little code.

However, all this patch adds is a test that fails: gcc built with gcc is
not bit-for-bit the same as gcc built with clang.

Since then I have included the build-path-prefix-map patch, use -rpath
$ORIGIN instead of -rpath "-lib", switched to gcc-7.2.0.  Still, no
bit-reproducibility.

Of course, Guix has the additional problem of the install-prefix that
debian and others do not have.

I'll be sending updated patches soon.

janneke.

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: avoid wrapper scripts when possible

[Ludovic Courtès]

Hi!

Hartmut Goebel  skribis:

> answering on a mail which Ricardo CCed to me and which did not make it
> to the list yet. Thus I full-quote.
>
> Am 04.11.2017 um 23:30 schrieb Ricardo Wurmus:
>> How about this:
>>
>> --8<---cut here---start->8---
>> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile
>> #!# (let (($0 (car (command-line (execl 
>> "/home/rekado/.guix-profile/bin/python3" $0 $0))
>> #!/home/rekado/.guix-profile/bin/python3
>> import sys; print("hello from python: "+sys.argv[0])
>> --8<---cut here---end--->8---
>>
>> The first two lines are Guile code, but they are also line comments in
>> shell, Perl, Python, Ruby, and R.  The Guile code in this example calls
>> the script again as a Python script.  Before doing that it can set
>> environment variables, like so:
>>
>> --8<---cut here---start->8---
>> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile
>> #!#
>> #\- (setenv "PYTHONPATH" (string-append "/gnu/store/foo:/gnu/store/bar:" 
>> (getenv "PYTHONPATH")))
>> #\- (let (($0 (car (command-line (execl 
>> "/home/rekado/.guix-profile/bin/python3" $0 $0))
>> #!/home/rekado/.guix-profile/bin/python3
>> import sys; print("hello from python: "+sys.argv[0])
>> print(sys.path)
>> --8<---cut here---end--->8---
>>
>> Did I overlook something?  Or could we use this hack to remove separate
>> wrapper scripts for Perl, Python, Ruby, and R?

Overall this looks good to me (with the adjustments Hartmut mentions).

Now, we should have a white list of languages for which this can be
done.  ‘wrap-program’ would grab the shebang and check if the
basename of the interpreter is in that white list.

Thoughts?

Ludo’.

Re: NFS mounts

[Ludovic Courtès]

Hi,

Konrad Hinsen  skribis:

>> By default, file systems are automatically mounted at boot time.  To
>> avoid that, you must add:
>>
>>   (mount? #f)
>
> That works well indeed. But I actually want that NFS filesystem mounted
> at boot time, ideally (not strictly required though).
>
> Of course, the real problem is that I cannot mount it at all because
> rpc.statd is missing.

Ooh, got it.  Well we could (ab)use the ‘dependencies’ field of
‘file-system’ to introduce a dependency on the rpc.statd daemon startup.

Ludo’.

Re: avoid downloading bootstrap binaries?

[Ludovic Courtès]

Hello,

Ricardo Wurmus  skribis:

> we have some Makefile rules that download some bootstrap binaries for
> us.  Why is this necessary?  Could we ship these bootstrap binaries
> instead?

These makefile rules were here mostly so that the tarball is not too big.

They are gone now in ‘core-updates’:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4f024ef3181c5235ff11789e8c8f62722f974787

See  for background.

HTH!

Ludo’.

Re: Status update on reproducible builds in Guix

[Gábor Boskovits]

Yesteday we had a discussion about that on irc.
Here it goes:


[15:15:16]  hello guix!
[15:16:01]  do we have a proposed way to build pyc files
reproducibly?
[15:16:50]  I've read in the report, that we are not there yet, but
is someone working on it?
[15:17:58]  g_bor: This is the report you mention? <
https://bugs.gnu.org/22533>
[15:18:10]  I'm not sure if anyone has been working on it since the
last message
[15:20:26] * Guest74 has joined #guix
[15:23:05]  thx, just what i was looking for.
[15:23:26]  It's possible that some work in this area is pending on
the 'core-updates' Git branch, but I'm not sure
[15:37:41]  At this stage we might as well wait for this to land
upstream: https://www.python.org/dev/peps/pep-0552/

So, it seems, that we are waiting for this pep to land upstream.


2017-11-05 16:49 GMT+01:00 Ludovic Courtès :

> Jan Nieuwenhuizen  skribis:
>
> > Ludovic Courtès writes:
> >
> >> Here’s an update on reproducibility in Guix:
> >>
> >>   https://www.gnu.org/software/guix/news/reproducible-builds-
> a-status-update.html
> >
> > At least 78% to possibly 91% reproduciblility of packages is not bad.
> >
> > Is there a (small) core that is already 100% reprocucible, like the
> > installation binaries/USB installer, bare-bones.tmpl or even
> > lightweight-desktop.tmpl?
>
> Good question!  I think as soon as you have Python .pyc files in the
> dependency graph (reference graph), there are non-reproducible bits.  We
> certainly have Python stuff in the base system reference graph, so
> that’s one thing we should fix most urgently I guess.
>
> Ludo’.
>
>

Re: Status update on reproducible builds in Guix

[Hartmut Goebel]

Am 05.11.2017 um 20:04 schrieb Gábor Boskovits:
> [15:37:41]At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/

Bad news: PEP 552 proposes a new file-format for .pyc file, so this
change can not be back-ported to Python 3.6 and older.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Status update on reproducible builds in Guix

[ng0]

Gábor Boskovits transcribed 5.3K bytes:
> Yesteday we had a discussion about that on irc.
> Here it goes:
> 
> 
> [15:15:16]  hello guix!
> [15:16:01]  do we have a proposed way to build pyc files
> reproducibly?
> [15:16:50]  I've read in the report, that we are not there yet, but
> is someone working on it?
> [15:17:58]  g_bor: This is the report you mention? <
> https://bugs.gnu.org/22533>
> [15:18:10]  I'm not sure if anyone has been working on it since the
> last message
> [15:20:26] * Guest74 has joined #guix
> [15:23:05]  thx, just what i was looking for.
> [15:23:26]  It's possible that some work in this area is pending on
> the 'core-updates' Git branch, but I'm not sure
> [15:37:41]  At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/
> 
> So, it seems, that we are waiting for this pep to land upstream.

PEP Status: Accepted.
I'm not familar with PEPs, other than that their sometimes very
long discussions on the mailinglists. Does this mean that the
missing pieces now are:

implementation
inclusion of this implementation in a new python release

> 2017-11-05 16:49 GMT+01:00 Ludovic Courtès :
> 
> > Jan Nieuwenhuizen  skribis:
> >
> > > Ludovic Courtès writes:
> > >
> > >> Here’s an update on reproducibility in Guix:
> > >>
> > >>   https://www.gnu.org/software/guix/news/reproducible-builds-
> > a-status-update.html
> > >
> > > At least 78% to possibly 91% reproduciblility of packages is not bad.
> > >
> > > Is there a (small) core that is already 100% reprocucible, like the
> > > installation binaries/USB installer, bare-bones.tmpl or even
> > > lightweight-desktop.tmpl?
> >
> > Good question!  I think as soon as you have Python .pyc files in the
> > dependency graph (reference graph), there are non-reproducible bits.  We
> > certainly have Python stuff in the base system reference graph, so
> > that’s one thing we should fix most urgently I guess.
> >
> > Ludo’.
> >
> >

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://dist.ng0.infotropique.org/dist/keys/
https://www.infotropique.org https://ng0.infotropique.org


signature.asc
Description: PGP signature

Re: 01/01: gnu: itstool: Update to 2.0.4.

[Kei Kebreau]

Marius Bakke  writes:

> Kei Kebreau  writes:
>
>> kkebreau pushed a commit to branch master
>> in repository guix.
>>
>> commit 13fbd174b5ffe5c2cc59e637f7859d357ab33d97
>> Author: Kei Kebreau 
>> Date:   Thu Nov 2 15:33:08 2017 -0400
>>
>> gnu: itstool: Update to 2.0.4.
>> 
>> * gnu/packages/glib.scm (itstool): Update to 2.0.4.
>> [inputs]: Replace python and python-libxml2 with python-2 and
>> python2-libxml2.
>
> This update broke "gtk-doc", so I reverted it.  Can you look into it?
>
> `itstool` is actually segfaulting:
>
> /gnu/store/kpxi8h3669afr9r1bgvaf9ij3y4wdyyn-bash-minimal-4.4.12/bin/bash:
> line 4: 2798 Segmentation fault itstool -m "${mo}"
> ${d}/C/index.docbook ${d}/C/fdl-appendix.xml
> make[2]: *** [Makefile:523: de/de.stamp] Error 139
> make[2]: *** Waiting for unfinished jobs
> /gnu/store/kpxi8h3669afr9r1bgvaf9ij3y4wdyyn-bash-minimal-4.4.12/bin/bash:
> line 4: 2808 Segmentation fault itstool -m "${mo}"
> ${d}/C/index.docbook ${d}/C/fdl-appendix.xml
> make[2]: *** [Makefile:523: fr/fr.stamp] Error 139

The itstool developers are aware of this problem [0]. The latest itstool
branch builds gtk-doc successfully, but I will wait until the issue is
closed before committing another patch.

[0]: https://github.com/itstool/itstool/issues/17
From 5d08f10d184a661d0ee5ddf13cd894d9299a9e1b Mon Sep 17 00:00:00 2001
From: Kei Kebreau 
Date: Sun, 5 Nov 2017 15:20:50 -0500
Subject: [PATCH] gnu: itstool: Update to 2.0.4.

* gnu/packages/glib.scm (itstool): Update to 2.0.4.
[source]: Apply patch.
[inputs]: Replace python and python-libxml2 with python-2 and python2-libxml2.
* gnu/packages/patches/itstool-fix-segfault.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk|  1 +
 gnu/packages/glib.scm   | 10 +++--
 gnu/packages/patches/itstool-fix-segfault.patch | 50 +
 3 files changed, 57 insertions(+), 4 deletions(-)
 create mode 100644 gnu/packages/patches/itstool-fix-segfault.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index dbfe6829e..b45499153 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -744,6 +744,7 @@ dist_patch_DATA =   
\
   %D%/packages/patches/ilmbase-fix-tests.patch \
   %D%/packages/patches/intltool-perl-compatibility.patch   \
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch\
+  %D%/packages/patches/itstool-fix-segfault.patch  \
   %D%/packages/patches/jacal-fix-texinfo.patch \
   %D%/packages/patches/java-powermock-fix-java-files.patch \
   %D%/packages/patches/jbig2dec-ignore-testtest.patch  \
diff --git a/gnu/packages/glib.scm b/gnu/packages/glib.scm
index bc8775004..71c21bd64 100644
--- a/gnu/packages/glib.scm
+++ b/gnu/packages/glib.scm
@@ -395,19 +395,21 @@ The intltool collection can be used to do these things:
 (define itstool
   (package
 (name "itstool")
-(version "2.0.2")
+(version "2.0.4")
 (source (origin
  (method url-fetch)
  (uri (string-append "http://files.itstool.org/itstool/itstool-";
  version ".tar.bz2"))
  (sha256
   (base32
-   "0fh34wi52i0qikgvlmrcpf1vx6gc1xqdad4539l4d9hikfsrz45z"
+   "0q7b4qrc758zfx3adsgvz0r93swdbxjr42w37rahngm33nshihlp"))
+ (patches
+  (search-patches "itstool-fix-segfault.patch"
 (build-system gnu-build-system)
 (inputs
  `(("libxml2" ,libxml2)
-   ("python2-libxml2" ,python2-libxml2)
-   ("python-2" ,python-2)))
+   ("python-libxml2" ,python-libxml2)
+   ("python" ,python)))
 (arguments
  '(#:phases
(modify-phases %standard-phases
diff --git a/gnu/packages/patches/itstool-fix-segfault.patch 
b/gnu/packages/patches/itstool-fix-segfault.patch
new file mode 100644
index 0..ed34aa18b
--- /dev/null
+++ b/gnu/packages/patches/itstool-fix-segfault.patch
@@ -0,0 +1,50 @@
+This patch comes from
+https://github.com/itstool/itstool/commit/9b84c007a73e8275ca45762f1bfa3ab7c3a852e2#diff-d15c1935d231593da203d653eef729ff.
+
+diff -ur a/itstool.in b/itstool.in
+--- a/itstool.in   2017-10-09 12:00:24.0 -0400
 b/itstool.in   2017-11-05 15:10:45.057031963 -0500
+@@ -477,6 +477,7 @@
+ if load_dtd:
+ ctxt.loadSubset(1)
+ if keep_entities:
++ctxt.loadSubset(1)
+ ctxt.ctxtUseOptions(libxml2.XML_PARSE_DTDLOAD)
+ ctxt.replaceEntities(0)
+ else:
+@@ -1043,6 +1044,7 @@
+ if self._load_dtd:
+ ctxt.loadSubset(1)
+ if self._keep_entities:
++ctxt.loadSubset(1)
+ ctxt.ctxtUseOptions(libxml2.XML_PARSE_DTDLOAD)
+ ctxt.replaceEntities(0)
+ else:
+@@ -1069,7 +1071,9 @@
+ ph_node = msg.get_placeholder(child.name).node
+ if self.ha