On Wed, 26 Feb 2025 05:35:29 +0800,
Sharlatan Hellseher wrote:
>
> [1 <text/plain (quoted-printable)>]
>
> Hi Guix,
>
> After reading the thread:
> How to move forward about Rust? antioxidant, cargo2guix, etc.
> <https://lists.gnu.org/archive/html/guix-devel/2025-02/msg00417.html>
>
> I've faced with the fact that Golang had a very close approach during
> compilation as Rust, where each package defined in Guix is not reusable
> as library but re-compiled from scratch with provided source.
>
> Some upstream news:
> - Go Toolchains, starting from go@1.21 https://go.dev/doc/toolchain
> - Go workspaces (aka monorepo), starting from go1.18
> https://go.dev/blog/get-familiar-with-workspaces
> - Go.mod https://go.dev/doc/modules/gomod-ref
> > The go directive sets the minimum version of Go required to use this
> > module. Before Go 1.21, the directive was advisory only; now it is a
> > mandatory requirement: Go toolchains refuse to use modules declaring
> > newer Go versions.
> - Golang supports way to many environment variables than current
> go-build-system is aware of
> https://raw.githubusercontent.com/golang/go/b38b0c0088039b03117b87eee61583ac4153f2b7/src/internal/cfg/cfg.go
>
> Some long waited discussions to bring go.mod aware build system and
> re-usable builds:
> - [PATCH 0/3] go-build-system and GOPATH improvements
> https://issues.guix.gnu.org/50227
> - [PATCH 0/1] Fix cross-compilation of packages that use go-build-system
> https://issues.guix.gnu.org/51981
> - [PATCH 2/4] guix: add go module aware build system
> https://issues.guix.gnu.org/74374
> - Golang build cache and content-based staleness in Guix
> https://issues.guix.gnu.org/32919
>
> Simple questions to ask:
> - is it save to build package A which requires package B@1 but with B@2
> - is it save to build package A which requires package B@2 but with B@1
> - is it save to build package A which requires go@1.18 with go@1.21+
> - is it save to build package A which requires go@1.23 with go@1.21
> - is it save to build package A which go@1.21 and deferent toolchain
> version (something new here...)
> - how to re-use the builds and save some N+ time and M+ energy on
> compilation
> - how to proper package multi modules repository, it's hard right now
> - if we follow the go.mod way, do we need to package each possible
> version to cover any combinations
>
> From my experience, go.mod is bumped just blindly on cron with services
> provided by GitHub or third parties and does not affect too much the
> compilation, until the upstream changes API dramatically; unit tests
> block this type of regression, that why ""#:tests? #t" is pleasurable
> with "./..." option, which is default now on go-build-system
>
> I might deep dive into GCD and prepare a fresh go-build-system
> adjustments proposal covering most of the concerns
I think the approach I'm currently experimenting with for our Rust ecosystem can
be applied to Go as well:
1. Import dependencies from lockfile as <origin>s
<origin> is capable of basic patching and unbundling, and provides a way to
deduplicate downloads.
This should remove most current <package>s, remaining ones will depend on all
dependencies in the lockfile.
--8<---------------cut here---------------start------------->8---
PACKAGE -> PACKAGE-cargo-inputs (<origin> <origin> ...)
--8<---------------cut here---------------end--------------->8---
(‘go.sum’ should contain all dependency information I think?)
Here[1] is an example of imported dependencies. Unused <origin>s are removed
by a script[2].
2. Only package complex dependencies
Module containing above <origin>s shouldn't depend on other package modules,
to avoid circular dependency.
When there's such need (e.g. some unbundling cases), involving dependencies
should still be made into a <package>.
To avoid duplication of imports, imported definitions of these dependencies
are redirected to the package ones.
Since it shouldn't contain package references, symbols are used, which will be
printed out in error message when accidentally used in inputs. e.g.
--8<---------------cut here---------------start------------->8---
(define rust-ring-0.17.11 'crates-crypto-rust-ring-0.17)
...
(define-public cargo-audit-cargo-inputs
(list ...
;; rust-ring-0.17.11
...))
--8<---------------cut here---------------end--------------->8---
--8<---------------cut here---------------start------------->8---
(define rust-openssl-src-300.4.2+3.4.1 'do-not-package-rust-openssl-src)
...
(define-public rust-cargo-c-cargo-inputs
(list ...
;; rust-openssl-src-300.4.2+3.4.1
...))
--8<---------------cut here---------------end--------------->8---
The comment is used to preserve reference for cleanup script (currently
implemented in ‘grep -wc’).
3. Make the build system aware of Go inputs and unpack them
This should be considered, with this model #:import-path and #:unpack-path are
unavailable.
Module support can be implemented too for the new model.
The build system should also have a phase like in cargo-build-system to check
pregenerated files.
(1.) and (2.) is the core to speed up the packaging process: Partially follow
upstream practice while keeping control of dependencies.
Some considerations are needed regarding dependency versions, Go is not as
convenient as Rust on upgrading and issues can't be just caught during
compilation it seems.
---
One issue I'm facing right now is, I want to save all imported <origin>s in one
module, like a registry, so that many efforts can be deduplicated. But this
approach has drawback in merging and rebasing.
Using multiple self-contained modules solves the issue but doesn't offer the
same benefits.
💡 How about making an actual registry? Add team-specific branchs dedicated to
maintain such modules and cleanup less often. Whenever there's change needed
for the main branch, it's first made into that branch and copied over. I think
with the speed up we can finish processing relevant patches within a timeframe
without breaking changes.
[1]:
https://git.boiledscript.com/hako/guix/src/branch/cargo/gnu/packages/rust-crates.scm
[2]:
https://git.boiledscript.com/hako/guix/src/branch/cargo/etc/teams/rust/cleanup-crates.sh
