Re: [Nix-dev] Avoiding threads in the daemon
Can't you unshare in the parent then setns back after fork? > On Dec 19, 2014, at 18:20, Eelco Dolstra wrote: > > Hi, > >> On 18/12/14 17:32, Ludovic Courtès wrote: >> >> Thus, I think Nix commit 49fe95 (which introduces monitor-fd.hh, which >> uses std::thread just for convenience) should be reverted, along with >> the subsequent commits to that file; then commit 524f89 can be reverted. > > I really don't want to get rid of threads because they're useful and I want to > use them more in the future (e.g. build.cc would be much simpler if it used > threads rather than the current event-driven approach; nix-daemon could handle > client connections with a thread rather than a process; etc.). > > I see a few ways to get PID namespaces back: > > * Do a regular fork followed by clone(... | CLONE_NEWPID | CLONE_PARENT) > (after > which the intermediate process can exit). > > * Call setuid/setgid via syscall() to bypass the locking in the Glibc > wrappers. > However, there might be other problematic functions so this is not a great > solution. > > * Get the Glibc folks to provide a way to run at-fork handlers with clone(). > > Clearly the first option is the easiest. > > -- > Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ > ___ > nix-dev mailing list > nix-...@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
Hey Ludo’,
Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.
Will look at your gcc and clang patches as well, thank you!
~Shea
Ludovic Courtès writes:
> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]:
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
> • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
> • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
> The comment in that file explains why the other options considered
> were unsuitable.
>
> • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
> • Likewise, we tell Clang where to find libc and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.
signature.asc
Description: PGP signature
Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
Hi Ludo’,
Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?
Thanks,
Shea
Ludovic Courtès writes:
> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]:
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
> • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
> • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
> The comment in that file explains why the other options considered
> were unsuitable.
>
> • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
> • Likewise, we tell Clang where to find libc and friends:
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>
> <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.
signature.asc
Description: PGP signature
Re: [Nix-dev] [PATCH] daemon: Add 'buildMode' parameter to 'buildPaths' RPC.
Can you resend with paths appropriate to the nix repo? There is no nix/ directory there. ~Shea On 2015-12-02 12:03, Ludovic Courtès wrote: * nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump to 0x10f. * nix/libstore/remote-store.cc (RemoteStore::buildPaths): Send the BUILDMODE when the daemon supports it. Reject invalid values of BUILDMODE for old daemons. * nix/nix-daemon/nix-daemon.cc (performOp) : Read the build mode when the client supports it. --- nix/libstore/remote-store.cc| 11 +-- nix/libstore/worker-protocol.hh | 2 +- nix/nix-daemon/nix-daemon.cc| 11 ++- 3 files changed, 20 insertions(+), 4 deletions(-) ___ nix-dev mailing list nix-...@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] /dev/shm inconsistency in chroot
On 01/24/2014 05:16 AM, Sree Harsha Totakura wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/23/2014 08:56 PM, Mark H Weaver wrote: We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. We can try creating a fixed set of device nodes, for example: /dev/null, /dev/random, /dev/urandom, /dev/sda etc. Has anyone tried this before? Another option is to mount a devtmpfs there, for systems which support it. Sree -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlLiPZUACgkQO2+K8UPCHzvyuwCgpPH4ndRBqFkITqbPcQ1UN4Ws JsYAniMrgj8mBvNMC7Jq1AkFv+bV/VUj =DbCU -END PGP SIGNATURE- ___ nix-dev mailing list nix-...@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] /dev/shm inconsistency in chroot
On 01/24/2014 04:17 PM, Ludovic Courtès wrote: Sree Harsha Totakura skribis: On 01/24/2014 06:13 PM, Mark H Weaver wrote: Shea Levy writes: Another option is to mount a devtmpfs there, for systems which support it. devtmpfs may give different devices on each machine and they may hinder our build reproducibility. OK. The thing is, we don't actually want most of the system's devices to be in the build environment, do we? These are all impurities. I don't think we want /dev/sda, for example. Sure, I agree. I propose we start enumerating commonly needed devices and create them. Sounds good. The major/minor device numbers may not be portable across OSes, which may be a problem for Nix, so that code may need to be #ifdef’d. Generally a good idea, but note that for now chroot is not enabled on non-Linux If in future, a package requires access to certain device while building (or during tests) we can include it in our list of created device nodes. Yes, but keep in mind that we’re not going to change that often, because it’s inconvenient. Thanks, Ludo’.
Re: [Nix-dev] GNU Guix 0.6 released
Hi Ludo, On Wed, Apr 09, 2014 at 08:29:25PM +0200, Ludovic Courtès wrote: > We are pleased to announce the release of GNU Guix version 0.6, > representing 597 commits by 16 people over 4 months. > Congratulations! > > This release comes with an updated QEMU virtual machine image that shows > preliminary work toward building a stand-alone GNU system with Guix. > The image uses the GNU Linux-Libre kernel and the GNU dmd init system, > and runs X11. It may be used primarily to try out Guix and dmd. > > > • About > > GNU Guix is the functional package manager for the GNU system > distribution. > > In addition to standard package management features, Guix supports > transactional upgrades and roll-backs, unprivileged package > management, per-user profiles, and garbage collection. Guix uses > low-level mechanisms from the Nix package manager, with Guile Scheme > programming interfaces. > > At this stage Guix can be used on top of an i686, x86_64, or mips64el > GNU/Linux system, or in a virtual machine. Future versions will > stand alone. > My brain keeps parsing this as "guix will run on bare metal", what does stand alone mean here? > > http://www.gnu.org/software/guix/ > > > • Download > > Here are the compressed sources and a GPG detached signature[*]: > ftp://alpha.gnu.org/gnu/guix/guix-0.6.tar.gz > ftp://alpha.gnu.org/gnu/guix/guix-0.6.tar.gz.sig > > ftp://alpha.gnu.org/gnu/guix/gnu-system-demo-0.6.qcow2.xz > ftp://alpha.gnu.org/gnu/guix/gnu-system-demo-0.6.qcow2.xz.sig > > Use a mirror for higher download bandwidth: > http://www.gnu.org/order/ftp.html > > Here are the SHA1 checksums: > > 2428bcf16051d8ac78d35285d947e583cd90c61a guix-0.6.tar.gz > 6dd9d1b4964ab6fa49ee6d49d826da6b7e2d99d3 gnu-system-demo-0.6.qcow2.xz > > [*] Use a .sig file to verify that the corresponding file (without the > .sig suffix) is intact. First, be sure to download both the .sig file > and the corresponding tarball. Then, run a command like this: > > gpg --verify guix-0.6.tar.gz.sig > > If that command fails because you don't have the required public key, > then run this command to import it: > > gpg --keyserver keys.gnupg.net --recv-keys EA52ECF4 > > and rerun the 'gpg --verify' command. > > This release was bootstrapped with the following tools: > Autoconf 2.69 > Automake 1.14.1 > Makeinfo 5.2 > > > To use the virtual machine image, run QEMU like this: > > qemu-system-x86_64 -enable-kvm -m 1024 -net nic,model=e1000 \ > -net user gnu-system-demo-0.5.qcow2 > > This enables networking support, which is useful when using Guix > inside of it. > > > • Changes since version 0.5 (excerpt from the NEWS file) > > ** Package management > *** Default store directory changed to /gnu/store > What was the motivation behind this change? Cheers, Shea > > Although it is still possible to configure Guix to use the old store > directory, we recommend that you start with a fresh install to /gnu/store. > > *** Substitutes from hydra.gnu.org must now be signed and authorized > *** Support for offloading builds to other Guix machines over SSH > *** New ‘guix archive’ command > *** New ‘guix system’ command; it can currently build VM images > *** ‘guix package’ accepts several arguments after -i, -r, and -u > *** Updating a profile with ‘guix package’ is now noticeably faster > *** ‘guix build’ can override a package’s source with the new > ‘--with-source’ > *** ‘guix build’ has a new ‘--no-build-hook’ option > *** ‘guix’ commands that build packages have a common set of options > *** ‘guix hash’ has a new ‘--recursive’ option > *** Daemon no longer bind-mounts the host’s /dev (improves reproducibility) > > ** Programming interfaces > *** New (guix pk-crypto) and (guix pki) modules for public key cryptography > *** New #:local-build? parameter for derivations that shouldn’t be offloaded > *** (gnu system vm) has procedures to build VMs that share the host’s store > *** New ‘text-file*’ procedure in (guix monads) > *** System service definitions are now provided by the (guix services) > modules > *** New X and SLiM services in (gnu services xorg) > *** New (guix git-download) and (guix svn-download) for git/svn checkouts > > ** GNU distribution > *** 91 new packages > > a2ps, asciidoc, aspell-dict-en, aspell-dict-eo, aspell-dict-es, > aspell-dict-fr, aumix, bitlbee, bogofilter, boost, calcurse, clusterssh, > corkscrew, cursynth, docbook-xml, docbook-xsl, elfutils, enscript, fftwf, > fltk, fuse, gcc-toolchain, giflib, glade, gmime, gnome-desktop, gnubg, > gnubik, > gnumach-headers, gnunet, gnuplot, gnurl, gphoto2, gtkglext, guile-json, > guile-reader-for-guile_2.0.11, gxmessage, hdf5, hurd-headers, inkscape, > iotop, > isc-dhcp, iso-codes, jnettop, libatomic-ops, libexif, libgphoto2, libmpcdec, > libmpdclient, libotr, libpcap, libpeas,
Re: [Nix-dev] [PATCH]
Hi Ludo’,
On Mon, Apr 14, 2014 at 03:38:01PM +0200, Ludovic Courtès wrote:
> The patch below allows files such as /nix/store/foo to be protected from
> GC if a root refers to them. This makes it easy for a build hook to use
> such files for its temporary working files.
>
> WDYT?
>
It seems a bit odd to me to be using the store as a temporary file
location. What's wrong with $TMPDIR?
>
> Thanks,
> Ludo’.
>
> diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc
> index 96e891f..299bf5f 100644
> --- a/src/libstore/gc.cc
> +++ b/src/libstore/gc.cc
> @@ -287,10 +287,7 @@ static void foundRoot(StoreAPI & store,
> const Path & path, const Path & target, Roots & roots)
> {
> Path storePath = toStorePath(target);
> -if (store.isValidPath(storePath))
Hm, don't we at least want to check that it's in the store at all?
>
> roots[path] = storePath;
> -else
> -printMsg(lvlInfo, format("skipping invalid root from `%1%' to
> `%2%'") % path % storePath);
> }
~Shea
> ___
> nix-dev mailing list
> nix-...@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] [PATCH] Allow roots to refer to arbitrary files in the store
On Mon, Apr 14, 2014 at 04:59:15PM +0200, Ludovic Courtès wrote: > l...@gnu.org (Ludovic Courtès) skribis: > > > Shea Levy skribis: > > > >> On Mon, Apr 14, 2014 at 03:38:01PM +0200, Ludovic Courtès wrote: > >>> The patch below allows files such as /nix/store/foo to be protected from > >>> GC if a root refers to them. This makes it easy for a build hook to use > >>> such files for its temporary working files. > >>> > >>> WDYT? > >>> > >> > >> It seems a bit odd to me to be using the store as a temporary file > >> location. What's wrong with $TMPDIR? > > > > Well, this is useful for build hooks, because it guarantees that any > > garbage they leave there will eventually be reclaimed. > > It also guarantees that this thing lives on the same file system as the > store, which means that rename(2) can be used to move it to its final > location in the store. > Hmm using the store as an auto-collected $TMPDIR seems wrong, but this is a decent argument. Do you have a specific use case already planned? > > Ludo’.
