Guidelines for bootstrapping with lots of inputs
Hi Guix! I've just managed to bootstrap the node-tape package using some of the helpers I introduced in 68941 (WIP). I want to use and reproduce the method I applied for phpunit in 67092, but it's a long method and I haven't had any input on it before now, so I'd like to at least validate the method before reproducing it. It consists in the following : - defining packages in an alphabetical order, in their final form (so using native-inputs that might not exist yet). - defining those potentially missing native inputs in a recursive way using package inheritance, by disabling tests and emptying native-inputs for all inputs. In the case of PHP, this only includes the PHPunit package, in patch 95 of 67092. In the case of node, I'll probably need to do that on several packages (tape, tap, typescript at least). There are some upsides compared to an approach where we explicitely define ...-bootstrap packages : - way less code - way more clarity about how the package is actually being bootstrapped. - we don't have to define packages twice, and that matters for PHP/Rust/JS where we have hundreds of packages for bootstrapping basic test native-inputs. But there are also big downsides : - stability in time : I don't have much worry due to centralization logic in PHP, not too much either for Rust, but in JS, bootstrapped packages for tape might gain typescript as an input, making regular headaches about how to bootstrap from there, if not using pinned versions. - computationally more intensive : updating a package could often mean rebuilding the whole bootstrap chain. This is also the case for unpinned inheriting ...-bootstrap packages. I've actually done a lot of bootstrapping with a more traditionnal approach with ...-bootstrap packages while working on the antioxidant/rust build-system and that is quite cumbersome. WDYT? Are there guidelines somewhere about how to do proper bootstrapping in these cases where hundreds of packages are involved? The actual inputs for PHPunit (patch 95 of 67092) : (inputs (letrec* ((package-list (list php-sebastian-version php-sebastian-type php-sebastian-cli-parser php-sebastian-recursion-context php-sebastian-resource-operations php-sebastian-object-enumerator php-sebastian-global-state php-sebastian-exporter php-sebastian-environment php-sebastian-diff php-sebastian-comparator php-sebastian-code-unit php-phpunit-php-timer php-phpunit-php-text-template php-phpunit-php-invoker php-phpunit-php-file-iterator php-phpunit-php-code-coverage php-phpspec-prophecy php-phar-io-version php-phar-io-manifest php-myclabs-deep-copy php-doctrine-instantiator)) (rewrite (lambda (p) (package (inherit p) (arguments '(#:tests? #f)) (native-inputs '()) (inputs (map (compose rewrite cadr) (package-inputs p))) (map rewrite package-list))) -- Best regards, Nicolas Graves
Re: Guidelines for bootstrapping with lots of inputs
> WDYT? Are there guidelines somewhere about how to do proper bootstrapping in > these cases where hundreds of packages are involved? Not really, but your idea to rewrite inputs for the bootstrapping process does simplify things a lot. How is typescript bootstrapped? Even though you'll have to repeat the trial and error process again, I think using sucrase is easier for bootstrapping typescript: https://lists.gnu.org/archive/html/guix-devel/2024-02/msg00122.html
Re: Guidelines for bootstrapping with lots of inputs
>How is typescript bootstrapped? Even though you'll have to repeat the trial >and error process again, I think using sucrase is easier for bootstrapping >typescript: https://lists.gnu.org/archive/html/guix-devel/2024-02/msg00122.html Nevermind, it's already bootstrapped with esbuild in guixrus.
Re: Guidelines for bootstrapping with lots of inputs
On 2024-02-20 15:11, Adam Faiz wrote: >> WDYT? Are there guidelines somewhere about how to do proper bootstrapping in >> these cases where hundreds of packages are involved? > > Not really, but your idea to rewrite inputs for the bootstrapping process > does simplify things a lot. > > How is typescript bootstrapped? Even though you'll have to repeat the trial > and error process again, I think using sucrase is easier for bootstrapping > typescript: > https://lists.gnu.org/archive/html/guix-devel/2024-02/msg00122.html Timothy claimed that building typescript with esbuild has been done (by singpolyma) in guixrus and it indeed seems it is. https://lists.gnu.org/archive/html/guix-devel/2024-02/msg00123.html https://git.sr.ht/~whereiseveryone/guixrus/tree/master/item/guixrus/packages/javascript/typescript.scm What I've done currently is to bootstrap tape (but not all packages using tape as a native-input in the process). The idea is to build tap and typescript from there. -- Best regards, Nicolas Graves
Trying to use guix-android
I added the guix-android channel[1] and did a guix pull. Then in a new terminal window I started guix repl, and followed the instructions in the readme. But I got a: guix repl GNU Guile 3.0.9 Copyright (C) 1995-2023 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guix-user)> ,use (android import repo) While executing meta-command: no code for module (android import repo) scheme@(guix-user)> (let ((manifest (get-manifest))) (fetch-manifest-repositories manifest)) ;;; :2:17: warning: possibly unbound variable `get-manifest' ;;; :3:3: warning: possibly unbound variable `fetch-manifest- repositories' ice-9/boot-9.scm:1685:16: In procedure raise-exception: error: get-manifest: unbound variable Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. scheme@(guix-user) [1]> So for now I fall back to repo and download from[2]. [1] https://framagit.org/tyreunom/guix-android [2] https://github.com/LineageOS/android.git
Re: Non-bootstrappable NPM packages
Nicolas Graves writes: > On 2019-07-24 15:41, Jelle Licht wrote: > >> Timothy Sample writes: >> >> [snip] >> >>> I’ve come to think that bootstrapping JavaScript might be easier than it >>> looks. As time goes on, Node gets better at the newer JavaScript >>> features. This removes the need for things like Babel or Rollup, since >>> with some care, Node can run the source directly with out any >>> transformations or bundling. That being said, TypeScript looks to be a >>> major issue, as it is used in many fundamental JavaScript packages and >>> it is not bootstrappable. >> >> Very recently (IE about 94 minutes ago), I found out something >> interesting that might be helpful; Sucrase[0] is, among other things, a >> typescript transpiler that does not do any type checking, and it only >> has some runtime dependencies. >> >> I created some “fiio”-packages as well [1] , and I have confirmed that >> it actually works! My next step was of course to compile TypeScript >> proper, and this worked with one tiny snag that I reported at [2]. After >> manually fixing these problems in the TypeScript source tree, I was able >> to transpile the TypeScript sources using guix-packaged >> `node-sucrase-bootstrap'. > > Hi Jelle! > > Did someone made some progress on the build-system since that to allow > for this to be taken into account? If you still have it, could you share > your "fiio" packages once again? The paste link is expired. Thanks! I don't have them anymore, but I could re-generate them. Are you still interested, or am I too slow in responding, and are your other node-related emails more relevant by now? KR, - Jelle
Re: Non-bootstrappable NPM packages
Hi there not sure if this was mentioned here but in case it wasn't — there was a proof of concept packaging of sucrase and jQuery as early as in 2022. You can see the relevant thread here[1]. So, yes, it's definitely easier than it seems (and easier than some used to describe it) — the only problem is that nobody seems to have picked up that work done by zamfofex ¯\_(ツ)_/¯ Wojtek [1] https://lists.gnu.org/archive/html/guix-devel/2022-11/msg00191.html -- (sig_start) website: https://koszko.org/koszko.html fingerprint: E972 7060 E3C5 637C 8A4F 4B42 4BC5 221C 5A79 FD1A follow me on Fediverse: https://friendica.me/profile/koszko/profile ♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ== ✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8= -- (sig_end) On Tue, 20 Feb 2024 20:04:33 +0100 Jelle Licht wrote: > Nicolas Graves writes: > > > On 2019-07-24 15:41, Jelle Licht wrote: > > > >> Timothy Sample writes: > >> > >> [snip] > >> > >>> I’ve come to think that bootstrapping JavaScript might be easier than it > >>> looks. As time goes on, Node gets better at the newer JavaScript > >>> features. This removes the need for things like Babel or Rollup, since > >>> with some care, Node can run the source directly with out any > >>> transformations or bundling. That being said, TypeScript looks to be a > >>> major issue, as it is used in many fundamental JavaScript packages and > >>> it is not bootstrappable. > >> > >> Very recently (IE about 94 minutes ago), I found out something > >> interesting that might be helpful; Sucrase[0] is, among other things, a > >> typescript transpiler that does not do any type checking, and it only > >> has some runtime dependencies. > >> > >> I created some “fiio”-packages as well [1] , and I have confirmed that > >> it actually works! My next step was of course to compile TypeScript > >> proper, and this worked with one tiny snag that I reported at [2]. After > >> manually fixing these problems in the TypeScript source tree, I was able > >> to transpile the TypeScript sources using guix-packaged > >> `node-sucrase-bootstrap'. > > > > Hi Jelle! > > > > Did someone made some progress on the build-system since that to allow > > for this to be taken into account? If you still have it, could you share > > your "fiio" packages once again? The paste link is expired. Thanks! > > I don't have them anymore, but I could re-generate them. > Are you still interested, or am I too slow in responding, and are your > other node-related emails more relevant by now? > > KR, > - Jelle > > pgphCMQ9QLveP.pgp Description: OpenPGP digital signature
February update on the Guile guix-daemon
Hey! Here's an overdue update on rewriting the Guix daemon in Guile, following on from the earlier thread on guix-devel [1] and the blog post [2]. 1: https://lists.gnu.org/archive/html/guix-devel/2023-09/msg00328.html 2: https://guix.gnu.org/en/blog/2023/a-build-daemon-in-guile/ Unfortunately I was quite slow to actually get stuck in with the code. Among other things, I've been distracted by trying to at least work around long running issues with the build coordinator and data service. I wanted to try and get QA back up and running and reduce the number of times that manual intervention is required, and this has been successful to some extent. I'm going to try harder to focus on the guix-daemon in the coming months however. With the time I have managed to spend working on the daemon though, I think I've made some progress. I started by getting a process to listen on a socket, then worked through implementing operations used by guix build (with many simplifying assumptions that is). This has given me some assurances about the viability of the technical plan I outlined in the blog post, as well as filling in some of the details. These are my current thoughts about the work ahead: - I've sent a couple of patch series [3][4] to adapt parts of Guix for use in the daemon, there'll need to be more changes in the database and substitute areas though. - I'm starting to understand, test and adapt the work on the guile-daemon branch, but there's more that needs doing on this. - Then there's the big areas to work on next: - I think I'm going to need to use thread pools for SQLite operations in the daemon, as the build coordinator does. - There's the low level work of setting up the build environment, the work on the guile-daemon branch helps a lot with this, but as pointed out by Ludo, there might be some issues with fork and similar operations in a Guile program using threads. - While I've implemented some of the server side protocol used by the daemon, I'd like to extract that code in to a module so that it's not wrapped up inside the daemon script 3: https://issues.guix.gnu.org/69291 4: https://issues.guix.gnu.org/69292 I think once I've made more progress in the above areas, I will hopefully have something that people could attempt to use in a limited capacity. Let me know if you have any comments or questions, Thanks, Chris signature.asc Description: PGP signature
Re: Non-bootstrappable NPM packages
On 2024-02-20 20:04, Jelle Licht wrote: > Nicolas Graves writes: > >> On 2019-07-24 15:41, Jelle Licht wrote: >> >>> Timothy Sample writes: >>> >>> [snip] >>> I’ve come to think that bootstrapping JavaScript might be easier than it looks. As time goes on, Node gets better at the newer JavaScript features. This removes the need for things like Babel or Rollup, since with some care, Node can run the source directly with out any transformations or bundling. That being said, TypeScript looks to be a major issue, as it is used in many fundamental JavaScript packages and it is not bootstrappable. >>> >>> Very recently (IE about 94 minutes ago), I found out something >>> interesting that might be helpful; Sucrase[0] is, among other things, a >>> typescript transpiler that does not do any type checking, and it only >>> has some runtime dependencies. >>> >>> I created some “fiio”-packages as well [1] , and I have confirmed that >>> it actually works! My next step was of course to compile TypeScript >>> proper, and this worked with one tiny snag that I reported at [2]. After >>> manually fixing these problems in the TypeScript source tree, I was able >>> to transpile the TypeScript sources using guix-packaged >>> `node-sucrase-bootstrap'. >> >> Hi Jelle! >> >> Did someone made some progress on the build-system since that to allow >> for this to be taken into account? If you still have it, could you share >> your "fiio" packages once again? The paste link is expired. Thanks! > > I don't have them anymore, but I could re-generate them. > Are you still interested, or am I too slow in responding, and are your > other node-related emails more relevant by now? I was going the esbuild route instead of the sucrase one, so that's not a need anymore. Both are viable however, this choice is questionnable. There are the packages from Wojtek in answer to this email that also bootstrap sucrase, so don't bother ;) I feel tests / native-inputs for Node is a bigger issue than the one of bundlers now. > > KR, > - Jelle > -- Best regards, Nicolas Graves
Re: Trying to use guix-android
no code for module (android import repo) Looks like you don't have guix-android after all. Can you check with guix describe? Le 20 février 2024 18:49:59 GMT+01:00, Tobias Alexandra Platen a écrit : >I added the guix-android channel[1] and did a guix pull. >Then in a new terminal window I started guix repl, >and followed the instructions in the readme. > >But I got a: > >guix repl >GNU Guile 3.0.9 >Copyright (C) 1995-2023 Free Software Foundation, Inc. > >Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. >This program is free software, and you are welcome to redistribute it >under certain conditions; type `,show c' for details. > >Enter `,help' for help. >scheme@(guix-user)> ,use (android import repo) >While executing meta-command: >no code for module (android import repo) >scheme@(guix-user)> (let ((manifest (get-manifest))) > (fetch-manifest-repositories manifest)) >;;; :2:17: warning: possibly unbound variable `get-manifest' >;;; :3:3: warning: possibly unbound variable `fetch-manifest- >repositories' >ice-9/boot-9.scm:1685:16: In procedure raise-exception: >error: get-manifest: unbound variable > >Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. >scheme@(guix-user) [1]> > >So for now I fall back to repo and download from[2]. > > > >[1] https://framagit.org/tyreunom/guix-android >[2] https://github.com/LineageOS/android.git >
