Jelle Licht <jli...@fsfe.org> writes: > Hello Guix, > > Seeing as this time next year I'll be finishing up my studies, this was a > now-or-never moment for me. I would love to spend this summer hacking on > both guix, and reading up on npm.
I think this project is a great idea! Feedback inline below. > #+AUTHOR: Jelle Licht, jli...@fsfe.org > #+DATE: 21-03-2016 > #+OPTIONS: toc:nil > > * Overview > > This project will allow Guix hackers to more easily package software that is > distributed through the Node Package Manager (npm), as well as allowing Node > developers on Guix to make use of the reproducible builds guarantee of Guix. > > After completing this project, it should be possible to easily make use of the > less-problematic packages in the npm registry on the Guix Software > Distribution. I assume that by “less-problematic” you are referring to the packages that actually upload their corresponding source code and not some minified blob. Maybe also the ones with the least insane dependency graph. Is this correct? > * Project structure > > Depending on findings in the early stages of the project, I foresee the > following distinct parts: > > 1. Extend Guix so it can 'simulate' the dependency graph generation of both > the > old and new npm [fn:6]. I don’t quite get this part. Why does it matter how old and new versions of NPM store dependencies? IIRC, there’s a NODE_PATH environment variable that we could set to specify where to find each library. Even if we have to make a similar directory structure as NPM for some reason, shouldn’t we just do what NPM 3 does and forget about older versions? > 2. Extend guix with an algorithm that matches npm's package.json flexible > version specification to a specific version. This sounds like it could be part of ’guix import npm’, but not in the Guix package recipes themselves. To date, we do not have a single importer that recursively imports packages, so this would be somewhat new territory. > 3. Add a ~guix import~ backend for the npm registry Yup. :) > 4. Package npm modules in guix Could you perhaps come up with an application or library to target for packaging that will exercise all of the code you will be adding to Guix? As you probably know, many non-trivial Node modules typically have *massive* dependency graphs (hundreds of nodes), so I think we need to be strategic in what we pick to package. I should also note that NPM, like other language package management systems, obscures the full dependency graph of a module by differentiating between “dependencies” and “development dependencies”. In Guix, we build from source, so we need those “development” (misnomer: they are really build-time dependencies, but apparently only a developer would want to build from source) dependencies, too. Ideally, we would like to run test suites for node modules as well. Unfortunately, npm packages are viewed more like binaries, so they strip out things like tests. It’s an open question if the tarballs available on npmjs.com are worth anything at all, or if we should instead build from the upstream source code releases. Unfortunately again, those upstream releases are usually just auto-generated tarballs from git tags on GitHub, which may prove problematic because no one actually uses them. Long story short, we need to be diligent about ensuring that we are truly building from source. :) > 5. Interface > > Right now, dependency resolution in npm is as stateful as can be, with even > the > installation order [fn:6] making a difference for where npm expects to find a > certain dependency. As two different dependency resolution mechanism are in > use, > of which especially the newer one is problematic because of its habit of > propagating dependencies upwards in the folder structure, both have to be > supported by a guix module. As I mentioned earlier, I don’t think that we need to support this at all. We should just arrange that NODE_PATH be configured correctly. We’re *replacing* NPM, after all, so we should use a more sane architecture if we can. Does that sound doable or are there technical reasons why it wouldn’t work? > npm uses SemanticVersioning range patterns [fn:7] to declare dependencies > between packages. A problem with this approach is that the same package > declaration can lead to an entirely different dependency graph, which defeats > the purpose of having a system with a focus on reproducible builds. If npm > packages are to be used, these version numbers need to be locked down to a > specific version, corresponding to the version that npm would install if left > to > its devices. A potential problem is that the entire dependency graph has to be > known ahead of time in order to pinpoint a 'correct' version of the > dependency. > A consistent ordering for 'installing' dependencies also has to be decided > upon. This is indeed a problem that needs solving *if* we consider recursive package imports a requirement for this project. Recursive imports haven’t been done before, so maybe this could be a “nice to have” part of the project rather than a “must have”. This functionality would be part of ’guix import npm’. > After the dependency resolution has been worked out, it should be possible to > create a guix import backend to leverage the code that has been produced up > till > now to allow the packaging of npm modules. The last part of the project > essentially serves as a starting point for packaging up useful npm packages. > > A stretch goal for the summer would be to create a guix build system for a > subset of npm packages, such as gulp. I would promote this from a stretch goal to a “must have”. One of the first steps of this project should be to create (guix build-system node) and (guix build node-build-system) modules that will do whatever the standard build+test+install process is for node modules. From what I’ve seen, a lot of node modules need gulp or grunt in order to build, so boostrapping one or both of those tools will probably be necessary in order to package anything substantial. > * Planning > > As I am currently a novice with regards to the internals of guix and the guile > programming language, up to the start of the actual project I will mostly be > reading up and hacking on guix. As such, the planning becomes: > > April 22 - May 22: > - Getting to know Guix(SD) > - Package programs using the guix import module > - An informal specification of the npm dependency resolution mechanism > - Getting the know the guix community and what everyone is working on > - Getting familiar with the contributing work flows > > May 23 - Jun 5: > - Formal specification to allow guix to simulate npm dependency resolution > (1) > > Jun 6 - Jun 19: > - npm Version pinning should be working (2) > - start working on the guix import backend > > Jun 20 - Jul 3: > - Guix import backend should be finished by now (3) > - Start testing npm packages > > Jul 4 - Jul 10: > - Holidays! > > Jul 11 - Aug 7: > - Solve any problems and corner cases with building and installing npm > packages (4) > > Aug 8 - Aug 23: > - If life goes a planned, Get all contributions ready to be merged back in > the > main Guix codebase. Please add to the timeline when you would expect to have node-build-system implemented. I may have some code laying around that could accelerate this a bit. > * About me > > My name is Jelle Licht, and I am currently studying Data Science at the > University of Technology Delft. I finished my BSc in Computer Science in 2015, > at the University of Technology Delft as well. Somewhere in the second > semester > of my studies, my roommate was using Vi in front of me, and since then I've > been > falling into the rabbit hole that is GNU/Linux, free software and the 'open > source' community. Somewhere along the line I became a huge fan of Clojure, > and > for a short while was a member of the very much unofficial Lisp Community > Delft. > > I have been a small-time contributor to some free software projects on github, > as well as having a day job to make ends meet using mostly Node. My online > handles include 'wordempire', as I am quite fond of reading, and 'jlicht'. For > the past month, I have been lurking on-and-off again in the #guix irc channel, > as well as reading up on some of the motivations behind reproducible research > and reproducible builds. > > Besides one week for visiting family and general holidays, this project would > be > my full time focus during the summer. Thank you for your proposal! Making JavaScript software packagable in Guix would be a *huge* advance in our quest for reproducible web applications. - Dave
