Hi Leo! Leo Prikler <leo.prik...@student.tugraz.at> writes:
> Hi Maxim, > > Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer: >> Hi Leo, >> >> Leo Prikler <leo.prik...@student.tugraz.at> writes: >> >> > Hello Maxim, >> > >> > As someone, who lobbied for the current status quo, I have some >> > thoughts to share. >> >> I'm happy you're commenting :-). >> >> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer: >> > > The Emacs packages built with the Emacs built system used to be >> > > installed in a sub-directory under the share/emacs/guix.d/ >> > > directory, >> > > but this was changed in commit >> > > 65a7dd2950ca13a8b942b2836260a2192351b271 >> > > shortly after having accommodated the site-start.el machinery to >> > > enable >> > > loading packages from any profile (via the EMACSLOADPATH search >> > > path >> > > specification). >> > Won't this reintroduce <https://bugs.gnu.org/38309> then? >> >> This bug was caused by the EMACSLOADPATH environment variable being >> too >> long. This was because the original search path specification was >> recursively adding any directories under site-lisp/ so that package >> installed under a guix.d sub-directory could were directly added to >> the >> load path. >> >> In the current situation, the EMACSLOADPATH contains only the site- >> lisp/ >> directory of any profile, plus the versioned lisp/ directory of the >> installed Emacs package. This patch series doesn't change that, so >> no, >> that bug wouldn't be reintroduced. > Which is also the reason why our startup code changes and people have > to accommodate to that, right? Yes. The startup code is now responsible to activate packages found not directly in the EMACSLOADPATH (share/emacs/site-lisp), but in sub-directories under share/emacs/site-lisp/guix. >> > > While this change allowed to expose simply and directly the >> > > packages >> > > found in EMACSLOADPATH, it does introduce the risk of file name >> > > collisions when multiple Emacs packages are joined in the same >> > > profile, >> > > especially with Emacs packages increasing in complexity (e.g., >> > > using >> > > more than a single .el file!) and expecting to have both their >> > > sources >> > > and resources extracted under their own nested directory rather >> > > than >> > > as >> > > a flat collection (ELPA, MELPA). >> > > One recent example I stumbled on was attempting to use the >> > > emacs-yasnippet-snippets package along with emacs-elpy; both >> > > wanted >> > > to >> > > install a 'snippets' directory to share/emacs/site-lisp/snippets, >> > > collided and resulted in problems that prove difficult to >> > > understand. >> > I believe that to be a problem in those packages. Data should not >> > be >> > installed into share/emacs/site-lisp, but share/emacs/etc. See for >> > instance also emacs-telega, which – while not quite adhering to the >> > above – installs its data in share/emacs/{telega-contrib,telega- >> > data}. >> > >> > Regardless of what you intend to do with site-lisp otherwise, data >> > files should *not*, I repeat *not* be installed there. I do >> > believe >> > standardizing share/emacs/etc is the way to go, however. >> >> While I agree that it would be more elegant the way you propose, >> that's >> not the way Emacs packages have been standardized. So going against >> the >> sin--8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------start------------->8--- gle "content directory" (c.f. info "(elisp) Packaging Basics") >> would >> break many Elisp library assumptions about where there files are, >> causing more friction (thus work) to adapt them in Guix. The content >> directory of an Elisp package can contain any kind of files (code, >> images, etc.), according to info "(elisp) Multi-file Packages". > > I suppose said manual is perhaps slightly outdated. The doc/package.texi file of the Emacs git repository is actively maintained, although cited contents above hasn't changed much for the last 8 years. In any case if you look into package.el, you'll see that package contents are fetched as a single archived, and its content installed to a single directory. >> A package is either a “simple package” or a “multi-file package”. A >> simple package is stored in a package archive as a single Emacs Lisp >> file, while a multi-file package is stored as a tar file (containing >> multiple Lisp files, and possibly non-Lisp files such as a manual). > When was the last time, you've installed a tar to site-lisp? I also > feel as though the expected contents are limited very much to README, > COPYING and the like: > > $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \ > -not -name '*.el' \ > -and -not -name '*.elc' \ > -and -not -name '*.el.gz' > ~/.guix-profile/share/emacs/27.1/lisp/term/README > ~/.guix-profile/share/emacs/27.1/lisp/COPYING > ~/.guix-profile/share/emacs/27.1/lisp/README The lisp/ directory is for "the standard Lisp files that come with Emacs" (from info "(elisp)Library Search"). I don't think we can draw much comparison between these and user-installable packages from ELPA or MELPA. Also, just to be clear, the tar is not installed as-is; it is extracted and its contents are installed in the "content directory". Citing again '(elisp) Multi-file Packages': Prior to installation, a multi-file package is stored in a package archive as a tar file. The tar file must be named ‘NAME-VERSION.tar’, where NAME is the package name and VERSION is the version number. Its contents, once extracted, must all appear in a directory named ‘NAME-VERSION’, the “content directory” (*note Packaging Basics::). Files may also extract into subdirectories of the content directory. One of the files in the content directory must be named ‘NAME-pkg.el’. It must contain a single Lisp form, consisting of a call to the function ‘define-package’, described below. This defines the package’s attributes: version, brief description, and requirements. For example, if we distribute version 1.3 of the superfrobnicator as a multi-file package, the tar file would be ‘superfrobnicator-1.3.tar’. Its contents would extract into the directory ‘superfrobnicator-1.3’, and one of these would be the file ‘superfrobnicator-pkg.el’. If you want to have a clearer idea of how packages from ELPA and the likes are installed, you can have a peek into the 'package.el' file shipped with Emacs (spoiler: it's basically just extracting the contents of the package archive to a single directory -- see the `package-unpack' procedure). Here's an experiment I've done with a profile containing the whole of our current emacs-build-system based packages, with this current change, that checks for collision at the top level of a package installation directory: --8<---------------cut here---------------end--------------->8--- $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-manifest.scm $ find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2 -mindepth 2 -not -regex '.*\.elc?$' \ | awk -F '/' '{ if ($9 in packages) \ {printf("%s directory of %s package collides with that of package %s\n", $9, $8, packages[$9])} \ else {packages[$9] = $8} }' doc directory of modus-operandi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2 tools directory of unidecode-0.2-1.5502ada package collides with that of package company-cabal-0.3.0-1.62112a7 snippets directory of minitest-0.8.0-1.1aadb78 package collides with that of package elpy-1.35.0 test directory of realgud-1.5.1 package collides with that of package flycheck-haskell-0.8-2.32ddff8 snippets directory of yasnippet-snippets-0.23 package collides with that of package elpy-1.35.0 snippets directory of haskell-snippets-0.1.0-0.07b0f46 package collides with that of package elpy-1.35.0 snippets directory of rspec-1.11-1.66ea7cc package collides with that of package elpy-1.35.0 doc directory of modus-themes-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2 data directory of emojify-1.2 package collides with that of package unidecode-0.2-1.5502ada test directory of systemd-mode-1.6 package collides with that of package flycheck-haskell-0.8-2.32ddff8 lib directory of slime-2.26.1 package collides with that of package robe-0.8.2 contrib directory of sly-1.0.0-7.68561f1 package collides with that of package slime-2.26.1 lib directory of sly-1.0.0-7.68561f1 package collides with that of package robe-0.8.2 doc directory of modus-vivendi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2 doc directory of evil-1.14.0 package collides with that of package racket-mode-0.0.2-6.5eb31a2 data directory of all-the-icons-4.0.1 package collides with that of package unidecode-0.2-1.5502ada snippets directory of feature-mode-20190801-1.11ae167 package collides with that of package elpy-1.35.0 --8<---------------cut here---------------end--------------->8--- So 17 Emacs packages in Guix currently conflict, and Guix seems to be silent about it when building a profile with them via 'guix package -p' (a bug?). > Perhaps we should ask Emacs folks how they feel about separating code > and data, but bugs have already been caused here and elsewhere by > stuffing them together. (I believe yasnippet was once already the > culprit at some point before it got reorganized.) In the meantime, > yes, doing so might cause extra work, but > 1) it only affects a subset of packages, and > 2) of this subset, we can prioritize those, that do exhibit this > behaviour. The above does output does show that's it's a subset of the packages that are affected. [...] >> We have two schemes to accommodate for our Emacs packages: >> >> 1. Those installed via their own mean, e.g. make install, using the >> gnu-build-system for example. These would still typically install >> their >> packages directly under site-lisp, possibly multiple files (that >> could >> still collide). > Why? Seems inconsistent, does it not? They can be adapted in time; until then it's easy to accommodate both. >> 2. Those installed via the emacs-build-system. With the proposed >> changes, those now go to site-lisp/guix/. The 'guix' sub-directory >> makes it unambiguous that anything found under is to be loaded by >> package.el; the `package-directory-list' variable can be pointed to >> it >> to have the Emacs' package library discover these self-contained >> packages. > I think you've lost me here. Basically, instead of being able to > `(require 'my-package)` you first need to load my-package through > package.el and then can `(require 'my-package)`? I am not sure, what > the benefit of that would be, if any. Only if you've specified --no-site-start, which would disable user-installed package discovery. This is the drawback of the trade-off, not the benefit :-). >> Currently if you use -Q, the Elisp libraries are in the load-path, >> but >> not loaded (you need manually do M-x load-library before you can use >> it), or call M-x guix-emacs-autoload-packages to load their >> autoloads. >> For programs, this mean (require 'some-library) works, but M-x >> some-library-autoloaded-function doesn't. >> >> After this change, if you use -Q the new style Emacs packages are not >> visible at all until you activate them with 'M-x load-library >> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix- >> emacs) >> then (guix-package-initialize), programmatically. > I do think, that there are legitimate reasons to not require a full > initialization here, particularly for the use in batch scripts, but I'm > not quite sure how much work we currently put into making sure that > everything is available. That's a fair point, although I'm not concerned about the overhead of loading compiled autoloads files. >> I don't see this as a problem. -Q is simply an alias for >> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources" >> "--no-site-file". > But not "--no-load-path" "--no-nothing" ;) One could argue that the spirit of -Q is to give an Emacs as minimal as possible (that's what the Emacs manual has to say about it, and the implicit --no-site-lisp has the meaning of "no user installed packages"). > Being able to load the same libraries as without is vital for debugging > and scripts. (Granted, some distros probably break with --no-site- > lisp, but that's nothing to strive for imo.) I think you meant --no-site-file here. There's not much magic; if you allow it to run, user installed packages are added to the load path and autoloaded; if you don't you don't then you only get Emacs standard libraries. Nothing else happens in the site-start.el file in Guix proposed here. Scripting works well enough that our 900'ish packages can be rebuilt with the change, many of which come with a test suite. Thanks for having me think harder about if this is really desirable/needed. For me the main plus is that we'd be adhering to the current standards used for Emacs packaging: everything is unpacked into a single directory owned by that package, so as long as we include any needed resources in the #:include argument of the emacs-build-system, it works as upstream intended it, with no need to patch variables or anything. I expect this situation to worsen as Emacs packages tend to get more complicated, and I don't feel strongly enough about it to go arguing about Emacs packaging standards on the Emacs mailing list :-). On the minus side it makes the startup slightly more expensive in terms of pure scripting, and requires the users to be mindful that site-start.el evaluation is needed for the user installed packages to be activated. Further thoughts? Thanks again, Maxim
