Idea for packaging rust apps
Hello, I hope this is the right place for this, apologies if it isn't. I'm working with a friend on a cargo importer that lowers the entry barrier and the maintainability costs for packaging rust apps in general, without sacrificing Guix dependency tracking and reproducibility of rust packages. When you get used to the tool, you can pretty much package rust apps with all the dependency chain very easily (I just packaged [1] texlab for my channel earlier this morning in less than 5 minutes, and i can easily update apps to the latest version in less than a minute). Progress is being tracked in [2] if anyone wants to check it out or contribute to it. It is currently missing a lot of features, but we hope to improve the user experience of the tool in the near future. It is a very simple tool, it essentially parses the Cargo.lock file and extracts all the relevant information for constructing the rust package definitions of all the cargo-inputs and the package itself, and outputs to stdio a guile module containing all the needed cargo-input chain as guix packages, with all the cargo-inputs self-contained and versions all sorted out for a working final package build. This way a packager only needs to focus on the actual package they are trying to build, instead of worrying about its cargo-inputs. Due to how cargo-inputs are organized in gnu/packages/crates-*.scm, and some current packaging guidelines for crates on Guix, we cannot simply contribute these self-contained packages generated directly from the Cargo.lock, thus requiring to use the guix crate importer and spending a lot of time fixing dependencies and worrying about other packages breaking in the process. I would like to propose some discussion around a better way of organizing the rust packages and its cargo-inputs in (gnu packages) for building rust apps that only need sources as cargo-inputs: 1) Create a new directory at gnu/packages/rust/ in which a contributor can commit self-contained rust apps scm modules. 2.1) Add a new module at gnu/packages/rust/my-rust-app-1.scm 2.2) Add a new module at gnu/packages/rust/my-rust-app-2.scm 3) All package definitions inside gnu/packages/rust/*.scm which are source-only (#:skip-build? #t) should not be exported. 4.1) gnu/packages/crates-*.scm will not cease to exist, existing rust apps packages that have a Cargo.lock could gradually be migrated to the new organization 4.2) libs which need to be built can still live in gnu/packages/crates-*.scm 5) A (define-public my-rust-app-1 (@@ (gnu packages rust my-rust-app-1) my-rust-app-1)) or equivalent could be done in a (gnu packages category) module to export the rust app in the desired category. 6) Unlike nix (which also parses the Cargo.lock in the build system), we don't lose the ability to make snippets for sources this way. 7) For updating/maintaining a rust package defined this way, one would be able to simply re-run the guix tool, and replace the gnu/packages/rust/my-rust-app.scm file, only copying over the final relevant package definition for the rust app with its tweaks for building, and passing over the new cargo-inputs generated by the guix tool. I believe that by only changing the way things are organized and having a guix tool that generates self-contained package definitions from Cargo.lock, it would be possible to greatly improve the time required for contributing new rust apps packages and maintaining them on Guix. Things don't need to be the way I described here, these are just my initial thoughts after several failed attempts and wasted time trying to contribute rust apps to Guix, I'd like to discuss workarounds and if the benefits are greater than the disavantages for an approach like this. The tool we made works really well for packaging for our personal channels, I am very satisfied with how easy it is, and I think Guix could benefit a lot by adopting a similar approach. What am I missing here? Are there any disavantages to this approach? Anything that would break from it if adopted on Guix? Any questions or suggestions? Murilo [1] https://codeberg.org/look/saayix/commit/c7643943545d62baba80cccee1650ebf94362858 [2] https://git.sr.ht/~look/cargo2guix
Re: Idea for packaging rust apps
Hi MSavoritias, > I wanted to ask, are you also aware of the antioxidant effort? > https://notabug.org/maximed/cargoless-rust-experiments I was not aware of it at all, thank you for enlightening me about the effort. > I was wondering of the differences since your build system seems to > still be using cargo under the hood instead of rustc. Yes, you are correct. It really does nothing special except making packaging of rust apps for the end user extremely easy. This approach still uses the cargo-build-system, it is not a replacement for it, thus carrying all the cargo (and cargo-build-system) flaws along with it. The antioxidant-build-system, on the other hand, aims to mainly solve the inefficiencies of the cargo-build-system and cargo itself. After looking at some mail exchange on the antioxidant effort, it will be hopefully much nicer to make rust packages once the rust-build-system is merged, and it will be a much better approach for Guix than what I am currently suggesting. I believe my approach with cargo2guix and transitively generating guix package definitions from Cargo.lock will mainly target user's Guix channels because of the simplicity of the packaging process. While it is better than what we currently have on Guix, it is nowhere ideal for Guix as the antioxidant effort is, because of CI and the amount of packages Guix has. Thank you once again for making me aware of antioxidant and the future rust overhaul on Guix. -- Best regards, Murilo
Re: Securing personal forks (Was: Discussion with Codeberg volunteers)
Hi Felix and 45mg,
I came up with a quick dirty hack for solving my authentication problems on guix
forks and channel forks, and upon seeing this discussion I thought I'd share
some bits.
It's a very simple trick, only requires (roughly) 3 lines of code in
'guix/git-authenticate.scm'.
See [1] for the diff (in the guix-hacks branch), just search for the 'init auth
hack' commit (I rebase it all the time so its better if you search for the
commit as any links will get broken soon).
If you don't want to, it goes something like this:
--
...
(unless (or (member (openpgp-public-key-fingerprint signing-key)
(commit-authorized-keys repository commit
default-authorizations))
(member (openpgp-format-fingerprint
(openpgp-public-key-fingerprint signing-key))
'(" ")))
...
--
After I add my fingerprint to the 'or' condition, I also add my public key to
the 'keyring' branch of the fork [2].
After that we need to bootstrap it once with '--disable-authentication', and
after bootstrapping it once you can freely rebase with only '--allow-downgrades'
(or don't rebase at all, your call) and enjoy full authentication on your fork.
No need to mess with introductory commits, you can just keep guix' default one.
No need to edit '.guix-authorizations'.
It also works for authenticated channel forks, you just need to add your key to
the 'keyring' branch of the channel fork and it will work.
Just sharing this quick hack I use, its far from ideal but it works for me, and
seeing this discussion I thought it might be useful for someone in the meantime
:)
Best regards,
Murilo
[1] https://codeberg.org/look/guix/commits/branch/guix-hacks
[2]
https://codeberg.org/look/guix/commit/8e2902aabb14e93864920bac51a178aeedf8a563
Re: How to move forward about Rust? antioxidant, cargo2guix, etc.
On Thu Mar 6, 2025 at 2:24 PM -03, Efraim Flashner wrote: > On Wed, Mar 05, 2025 at 09:20:50PM +0800, Hilton Chain wrote: >> 1. Unpack package source then update lockfile: >> --8<---cut here---start->8--- >> cargo generate-lockfile >> --8<---cut here---end--->8--- >> >> 2. For adding new packages check output of ‘cargo audit’ and ‘cargo license’. > > These two were easy. I realized later that we could forgo running > 'cargo generate-lockfile' if we wanted to use the Cargo.lock that > already exists (if it does), but we probably want to make sure we have > more up-to-date dependencies as much as possible. I could see, as part > of a rust-team branch, going through and updating the lockfile every now > and then even if there is no update to the actual package, as the > dependencies get updated upstream. I think always having the most up-to-date dependencies is a good thing as well. Just something to keep in mind when doing this is that when we ship the same lock as the upstream release tag, we are making sure the final program will behave as closest as possible to the release version shipped upstream. This is a good thing for upstream because it can receive bug reports from guix users without much worry about the dependency chain being different and we are more likely to have the same issues as upstream has. So while I think the chances are low, there's always a chance of something breaking at runtime if we start bumping the locks. Even if the semver is compatible you never know how the final program might behave. Again, not against it, just something to keep in mind. >> 3. Update module >> --8<---cut here---start->8--- >> guix import --insert gnu/packages/rust-crates.scm crate --lockfile=<...> >> <...> >> etc/teams/rust/cleanup-crates.sh >> --8<---cut here---end--->8--- > > I got tripped up with this a bit. --lockfile=path/to/lockfile Also, not sure if its a shell thing, but I spent a couple minutes until I realised that passing '~' to --lockfile didn't work (it triggered the normal crate importer, without lockfile, instead) like so: --- Does not work as expected: $ guix shell -D guix -- ./pre-inst-env guix import \ --insert gnu/packages/rust-crates.scm crate \ --lockfile=~/path/to/Cargo.lock Works as expected: $ guix shell -D guix -- ./pre-inst-env guix import \ --insert gnu/packages/rust-crates.scm crate \ --lockfile=/path/to/Cargo.lock --- > I had 4 dependencies which needed git-fetch, and I packaged them in > rust-crates.scm also. I tried changing the names to > rust-xx-for-uv.tar.gz, but that didn't trick cargo into unpacking them. > In the end I had to add them as inputs and copy them over manually. One > of them, version-ranges, was inside a directory of another one, which > was annoying to figure out, and I think would've needed manual > intervention even if git sources were unpacked automagically. A bit of a hack I do is to use the forge archive links for these, for example: --- (origin (method url-fetch) (uri "https://github.com/pythonesque/shaderc-rs/archive/f2605a02062834019bedff911aee2fd2998c49f9.tar.gz";) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "14na50a585lb0pkdz3kraw2sgk8qjd78972aimcwdlkl9kyvvd0z"))) --- Then it already comes in '.tar.gz' and gets handled by cargo-build-system. Not the best, but it gets the job done and I think its easier than modifying the phases to add the extra sources. On Wed Mar 5, 2025 at 10:20 AM -03, Hilton Chain wrote: > My current workflow uses these packages: > --8<---cut here---start->8--- > guix shell rust rust:cargo cargo-audit cargo-license > --8<---cut here---end--->8--- > > 1. Unpack package source then update lockfile: > --8<---cut here---start->8--- > cargo generate-lockfile > --8<---cut here---end--->8--- > > 2. For adding new packages check output of ‘cargo audit’ and ‘cargo license’. > > 3. Update module > --8<---cut here---start->8--- > guix import --insert gnu/packages/rust-crates.scm crate --lockfile=<...> <...> > etc/teams/rust/cleanup-crates.sh > --8<---cut here---end--->8--- > > 4. Check module diff, content of sources marked as TODO and output of > 'check-for-pregenerated-files phase. I tried it on a couple packages, the importer seems to work really well, so far so good. > I added etc/teams/rust/rust-crates.tmpl in case multiple modules are wanted. > But I still hope you to try the one-module approach first :) I'm still a bit concerned about having too many git conflicts to solve when multiple patches from different people start coming a
Re: How to move forward about Rust? antioxidant, cargo2guix, etc.
Hi Simon, thanks for bringing up the discussion again. On Fri Feb 21, 2025 at 11:23 AM -03, Simon Tournier wrote: > How does it compare with antioxidant? And your (Nicolas) ideas? Antioxidant and Nicolas proposal is for a new build system (rust-build-system) in Guix best aligned with we already do for other languages, which is using libs shared and re-using previously pre-built stuff as inputs for a new package output. Using rust-build-system comes with a major advantage: - Lower build times, since we are only building every rust package once; - Meaning Cuirass will build rust packages much much faster than when using cargo; - As an indirect consequence, other packages in guix are evaluated faster. In contrast, my proposal is to continue using the cargo-build-system, but reorganize how we define and accept contributions for rust packages on (gnu packages *) by transitively generating guix package definitions from Cargo.lock. >From my perspective it makes contribution easier because you need only to generate the module contents, with, say, 'guix import' and send that isolated module. The module contents could be only origins, as Ludo' suggested, together with the actual package definition for the rust app you're packaging. It also makes the maintainability of the rust ecosystem in guix easier: - "I want to update the 'melon' package." (the current guix way) * I run 'guix refresh -u melon' to update the version and hash. * 'guix build melon' * Oh, guix has 'rust-orange@1.1.0' but it needs 'rust-orange@1.2.0'. * I'll just update 'rust-orange'! * But then I'll have to fix the 'strawberry' package that requires '= 1.1.0'. Notice that because rust packages use #:cargo-inputs there's no easy and documented way way to check the above, AFAIK, until cuirass complains about it. * I'll just make 'rust-orange-1' the new 1.2.0, and make a new 'rust-orange-1.1' for the 'strawberry' package. It's easier to do this instead of patching the Cargo.toml because you don't need to test if 'strawberry' works - a package that you never used before and have no idea what is, you're just trying to update the 'melon' package. When 'strawberry' updates, it will probably leave 'rust-orange-1.1' hanging. * Make sure every affected package builds (even libraries which IMO shouldn't be built). * Write multiple commit messages for each package change across diferent modules. - "I want to update the 'melon' package." (my proposed way) * 'melon' package is in 'gnu/packages/rust/melon.scm'. * I run 'guix refresh -u melon' to update the version and hash. * I run 'guix import cargo melon', it fetches the 'Cargo.lock' from the origin, and generates the dependencies. * I delete the entire 'gnu/packages/rust/melon.scm', except for the 'melon' package, and paste the new generated cargo-inputs. * Update 'melon' #:cargo-inputs to the newer generated ones. These steps could be automated with a tool (perhaps attached to 'guix refresh') to automatically do these trivial steps if we agree on sort of a pattern/template for organizing rust apps. * One single commit to the module update (because updates mostly won't affect other modules). >From my experience using this workflow on my personal channel I can package most rust apps in less than 10-15 minutes (including build time), and updates are really trivial to the point I can do even major version bumps in less than 2 minutes (excluding build time, and including some manual editor steps which could be automated). I'd like it too if we could get rid of #:cargo-inputs and use regular package inputs, as Efraim noted. I might be mistaken but I believe the new 'zig-build-system' does something similar to how cargo works when building, and it used to have #:zig-inputs in the WIP implementation which was rewritten later as regular input fields, perhaps we could do some analogous work for 'cargo-build-system'. Best regards, Murilo
Re: How to move forward about Rust? antioxidant, cargo2guix, etc.
On Wed Feb 26, 2025 at 12:30 PM -03, Efraim Flashner wrote: > I was looking closely at cargo2guix earlier this week, and I hacked up > that if a [[package]] had a source but no checksum it was a git > reference, and if it had no source and no checksum then it must be > inside the workspace. Yes, this is true. I've known this for a while but never had the motivation to implement some logic for it in cargo2guix. I've been removing the workspace cargo-inputs by hand until now - thanks for the patch by the way, I'll look at it later :) > Looking at guix/build/cargo-build-system, the "easiest" option would be > to take the git-checkout and then turn it into a .tar.gz. Otherwise > crate-src? and 'configure would need to be adjusted to handle git > checkouts. I think so too, while its easier it feels a bit hacky, but it should do the job for now. We could always do a proper support for git checkouts later. --- (A bit of a parenthesis) Perhaps its also a good time to add "--offline" to all cargo invokes to effectively prevent it from executing online operations. --- > Would it be easier to have 1 package per module, as in just the cargo > inputs for zoxide in gnu/packages/rust-crates/zoxide.scm, and then you > wouldn't need to worry about removing variables that aren't used by > zoxide anymore but are used by another package? I agree with Efraim here. My initial thoughts were to have one package per module not only because its easier to do, but also because it avoids any merge/rebase conflicts with other packages/patches.
Re: How to move forward about Rust? antioxidant, cargo2guix, etc.
On Wed Feb 26, 2025 at 3:43 AM -03, Hilton Chain wrote: > Time to try in your own channel!!! I just did, it worked for helix. I'll be trying in some more rust packages in the following days. > * Package cargo-audio[1] and cargo-license[2] or similiar software, and write > a > step-by-step packaging workflow. I like this idea of the step-by-step packaging and having perhaps a contribution workflow specifically for rust in the manual, we could also have a section and write some tips of some common problems people might encounter when packaging rust apps (for example "what to do when the package has a git reference in Cargo.toml?"). > * Package cargo-audio[1] and cargo-license[2] > > [1] https://github.com/rustsec/rustsec/tree/main/cargo-audit > [2] https://github.com/onur/cargo-license Attached is a package for cargo-license done with your implementation :) (PS: Took me less than 1 minute to package from an empty module) (define-module (saayix packages rust cargo-license) #:use-module ((guix licenses) #:prefix license:) #:use-module (gnu packages rust) #:use-module (guix build-system cargo) #:use-module (guix download) #:use-module (guix gexp) #:use-module (guix git-download) #:use-module (guix packages) #:export (cargo-license)) ;;; ;;; *-cargo-inputs and crates definitions ;;; ;;; This file is managed by ‘guix import’. DO NOT add definitions manually. ;;; (define cargo-license-cargo-inputs (delay (list rust-ansi-term-0.12.1 rust-anstream-0.6.11 rust-anstyle-1.0.4 rust-anstyle-parse-0.2.3 rust-anstyle-query-1.0.2 rust-anstyle-wincon-3.0.2 rust-anyhow-1.0.79 rust-camino-1.1.6 rust-cargo-platform-0.1.6 rust-cargo-metadata-0.18.1 rust-clap-4.4.18 rust-clap-builder-4.4.18 rust-clap-derive-4.4.7 rust-clap-lex-0.6.0 rust-colorchoice-1.0.0 rust-csv-1.3.0 rust-csv-core-0.1.11 rust-either-1.9.0 rust-equivalent-1.0.1 rust-getopts-0.2.21 rust-hashbrown-0.14.3 rust-heck-0.4.1 rust-indexmap-2.1.0 rust-itertools-0.12.0 rust-itoa-1.0.10 rust-memchr-2.7.1 rust-proc-macro2-1.0.78 rust-quote-1.0.35 rust-ryu-1.0.16 rust-semver-1.0.21 rust-serde-1.0.195 rust-serde-derive-1.0.195 rust-serde-json-1.0.111 rust-serde-spanned-0.6.5 rust-smallvec-1.13.1 rust-spdx-0.10.3 rust-strsim-0.10.0 rust-syn-2.0.48 rust-thiserror-1.0.56 rust-thiserror-impl-1.0.56 rust-toml-0.8.8 rust-toml-datetime-0.6.5 rust-toml-edit-0.21.0 rust-unicode-ident-1.0.12 rust-unicode-width-0.1.11 rust-utf8parse-0.2.1 rust-winapi-0.3.9 rust-winapi-i686-pc-windows-gnu-0.4.0 rust-winapi-x86-64-pc-windows-gnu-0.4.0 rust-windows-sys-0.52.0 rust-windows-targets-0.52.0 rust-windows-aarch64-gnullvm-0.52.0 rust-windows-aarch64-msvc-0.52.0 rust-windows-i686-gnu-0.52.0 rust-windows-i686-msvc-0.52.0 rust-windows-x86-64-gnu-0.52.0 rust-windows-x86-64-gnullvm-0.52.0 rust-windows-x86-64-msvc-0.52.0 rust-winnow-0.5.34))) (define rust-ansi-term-0.12.1 (let ((name "rust-ansi-term") (version "0.12.1")) (origin (method url-fetch) (uri (crate-uri "ansi_term" "0.12.1")) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "1ljmkbilxgmhavxvxqa7qvm6f3fjggi7q2l3a72q9x0cxjvrnanm") (define rust-anstream-0.6.11 (let ((name "rust-anstream") (version "0.6.11")) (origin (method url-fetch) (uri (crate-uri "anstream" "0.6.11")) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "19dndamalavhjwp4i74k8hdijcixb7gsfa6ycwyc1r8xn6y1wbkf") (define rust-anstyle-1.0.4 (let ((name "rust-anstyle") (version "1.0.4")) (origin (method url-fetch) (uri (crate-uri "anstyle" "1.0.4")) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "11yxw02b6parn29s757z96rgiqbn8qy0fk9a3p3bhczm85dhfybh") (define rust-anstyle-parse-0.2.3 (let ((name "rust-anstyle-parse") (version "0.2.3")) (origin (method url-fetch) (uri (crate-uri "anstyle-parse" "0.2.3")) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "134jhzrz89labrdwxxnjxqjdg06qvaflj1wkfnmyapwyldfwcnn7") (define rust-anstyle-query-1.0.2 (let (
