I rewrote this email 3 times already! I'm making a layout then. Sorry if the style is inconsistent.
1) About Antioxidant/rust-build-system Tremendous, back in the day I could rebuild half the whole rust world in about a day. Complexity was a thing however (definitely): you had to check the version, and then by trial and error you had all the #:features requires by any package in the chain for the final package, and then finally sometimes patch by hand... Just to notice that we need 5 different versions of the same package anyway. The perfect alternative would be the cargo that doesn't require rebuilds in its computing model, because calling the compilers like in antioxidant... is indeed low-level. IIRC, I didn't have cross-building with rust-build-system. To be fair, it is possible to maintain but requires more human time and expertise (although less compilation time). 2) About Workspaces A thing I did that was not in antioxidant and that is applicable to the current build system is to build whole workspaces instead of only building packages. Here workspace = a set of packages built together. The idea is that some entire files (crates-gtk for instance, but crates-crypto too, basically every -impl or -types package is in a workspace with its dependent package) are developped and built in the same repositories, but only released as different crates. Antioxidant (but I expect cargo to be able too) was able to build a whole workspace at once. I used multiple outputs, one for each crate to make that possible. I didn't do that, but if we want real packages from there, we can just use a copy-build-system or a trivial-build-system with a simple link with the source set to the crate (one of the outputs of the workspace). My crates-gtk.scm was only a hundred lines long IIRC. This would also help build times a bit, because although we rebuild a lot, if we produce 10 packages from a single build, we rebuild 10 times less. I did factor out my rust-workspace-build-system in rust-build-system with this simple observation : a package can be described as a workspace with a single package, so I didn't even needed to carry a special build-system. (It made some composition fun too: I could inject a few packages to build together in a virtually created workspace). It's also my objective to bring this idea to Node and possibly Go, which have the same concept of workspaces could make the maintenance burden significantly lower in Guix : we try to do too much compared to what these tools already do! I haven't tried workspaces for this though : the cases where we have circular dependencies and might require #:cargo-inputs and #:cargo-development-inputs in the first place might be handled well by cargo by putting all the problematic packages (as seen by Guix) in the same workspace, letting cargo handle the build, and then composing packages on Guix side with copies/links to the outputs. I don't know however if we could build a suitable solution for getting rid of these from there. Something like "if a circular dependency affects only packages with node/rust/go build-systems, put them all in the same workspace, and then separate them back on Guix side". 3) Current Rust in Guix situation is the worse than even in a classic distribution My point is however that "debundling" / the guix approach is a bad term for what currently happens in our crates. Truthfully, we most likely don't even care about rust intermediate crates (like 90% of our rust crates). We spend enormous efforts and energy trying to build and rebuild and rebuild crates that are only ever important in that they are a part of a user-facing binary or library. I would even argue provocatively that these 90% efforts are of no use to no one. For instance take the case of a dependency of 3 user-facing packages at depth 5 for the three, currently is it rebuilt at least 15 times (or possibly double with tests). Even in a classic linux distribution, it'll be rebuild multiple times, but only 3 times (or possibly double with tests). Let's face facts: either we wish to go the full guix way, either we don't and should have a different approach (Murilo's suggestion is one). 4) The full Guix way Most likely : we don't have the manpower in the short term for going the full guix way, because, and I second Murilo's point here, we dive in dependency hell quickly with more required efforts than reasonable (having implemented it myself, despite being quite proud of it). BUT: Maybe antioxidant is still doable. IIRC the hardest was not setting #:features correctly (I hope IRC). This is because the information about features required for parent packages is located in parent / dependent packages, not in child packages. Cargo chooses to rebuild everything ; but there is at least another solution for that : parse all the dependents of a package to record the different features/versions a package would need to be generated with. This means when importing, also sinking into crates.io huge database, but we could construct an inverse cache or something. And an importer could then generate more than one version of a package, but several variants, if several variants are indeed required to build all its transitive parents. To be even more precise we could search only in blessed.rs or lib.rs. But we need to pull that from Rust, with dependents too (and not in Guix like Antioxidant did). This could lead to a channel like we've seen for Emacs packages or CRAN, generating regularly from complete upstream info, and we could incorporate piece by piece packages that build in this channel into Guix. With limitations: the deduplication of packages (having 4-5 versions of the same package) will probably still be necessary. Otherwise there's just a lot of work by hand, we probably would like to avoid. 5) The classic distribution way, Guix flavored Combining the transitive package approach with the workspace approach, we could reach such a workflow for a package: - Extract all inputs from a Cargo.lock - If necessary, patch inputs in Guix (debundling) - Create a big workspace and inject all inputs inside (Guix side) (here, no need (because no benefit) in defining intermediary packages in Guix, just download sources). Not a whole file with hundreds of lines to update regularly like Murilo proposed, rather a single <package> we could bump like any other (as long as inputs' checksums are recorded somewhere). - Let cargo handle the "build" part, patching if necessary - Only extract the final output package we want users to have access to (binary or library) Sounds a lot like what any classic distribution would do :p I don't like the idea, but the environmental activist in me tells me it's a more sober way in both human and computer resources. As counter-intuitive it might seem, as long as cargo's build model and ecosystem doesn't change, I don't see a better way. We should still strive to check for binaries / nonfree code / bundled libraries in the inputs of course, replacing pure crate sources by our unbundled/patched versions, but not building them for the sake of it. The main responsibility of the rust team in this case would simply be to manage a huge file of checksums (could be automated), check that inputs are not corrupted and don't bundle external libraries, that licenses are OK and that some necessary patches (linking libraries) are well managed. That would make us delete 90% of our crate definitions, but if they are of no use to no one (this is provocative, but prove me wrong here :p), that is probably the right thing to do. We would basically keep rust-apps.scm, and any crate that is used in another package without being rebuilt. (I see now that tests might be a thing ; yes but I'm still pretty sure it's not worth the cost). Conclusion) I think we should go with (5) for now, maybe some motivated individual (not me this time, but I can make some guidance based on memory) will be able to tackle (4), rebase antioxidant, branch it to a rust package provider, get it to work automatically WITH FEATURES SUPPORT in a channel and then, once it works, at some point we ~should~ reconsider having another rust-build-system in Guix and slowly switch (from the ground up). But the promise of a potential guix-aligned rust-build-system should not hinder a proper rethink of the way we currently do things, and I think we should do (5) no matter what. Even deleting so hard-earned crates, they most likely won't be helpful in a (4) scenario. --- Below is a first version of my answer, after the discussion on workspaces and as a response to Murilo. While writing it, I stumbled on the obvious fact that it was still missing the point of the usefulness of such a "debundled mainline rust", since everything would still be rebuilt everywhere. Why bother bulding and maintaining for no benefit to no one? --- What about a mix of both ? (Haskell has the same issue too) Like having a proper guix way for "mainline rust" (as seen in here for instance : https://blessed.rs/crates or later if sucessful lib.rs), and forcing importers to pick up from there when a suitable version is present (developping version comparison is a need in Guix anyway, at least for everything related to recursive imports which are poorly handled now) (I mean with support for && || >= > < <= operations and conversions with semver notation), and we try and use rust workspaces in our "mainline rust", which should also handle everything that has a FFI/dependency in Guix. That way - we debundle the bulk with our "mainline rust" and bring down both maintainance burden and compilation times. - we give users an easier way to update their packages. - we could possibly even outsource what is not "mainline rust" neither user-facing applications we want in Guix to specialised channels (e.g. a guix-rust-past for old versions we don't support anymore). PS I see this only now but since you wish to build everything in a file and then update that file, you could as well... do it in a cargo workspace! i.e. Inject all inputs, let cargo do its thing, and have a single output. Only care in guix about the final package description/version and everything. No strings attached, no need to support hundreds of lines we don't care about! (I still vomit writing package descriptions for -impl crates). -- Best regards, Nicolas Graves
