Efraim Flashner <efr...@flashner.co.il> writes:
[[PGP Signed Part:Undecided]]
On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote:
Efraim Flashner <efr...@flashner.co.il> writes:
> On the other hand, by generating it during the build of each
> package we
> make sure to pull in all the crates which exist in the build,
> so we
> could add into a profile/manifest just the crates listed in a
> Cargo.toml
> and then each crate would pull in its own dependencies, and
> then the
> profile hook could combine them all together.
Thanks. Just to make sure I understand: it sounds like you're
saying that by
creating the JSON index files up front, we'd be preserving some
knowledge
about a package's dependency graph that isn't easily recovered
later by
recursively walking through inputs and cargo-inputs for the
package specs in
a manifest/profile?
If we create it upfront it may be easier to keep track of the
cargo-dependencies, but I suppose it would mostly depend on the
implementation. I'm not sure that'd actually be the case, so
don't
worry about it too much. Whatever we end up with will be better
than
what we have now, and we can always make it better later if it
needs to
change.
Hi Efraim, I spent some time on this and have a few thoughts
below. In short, I made progress but also hit some roadblocks that
give me second thoughts about the approach we discussed up-thread.
As you suggested, I kept the metadata generation code in
cargo-build-system and used the cargo-registry profile hook to
merge all of those package artifacts into index files. I also
modified the hook to lift the full closure of source dependencies
into the profile for any rust packages in the manifest, borrowing
logic from `(guix build-system cargo)`.
However, testing revealed some issues with this approach:
1) Formerly, for any cargo package with `#:skip-build? #t` the
crate was treated as an opaque file: it was copied to a location
where cargo could access it, but otherwise unprocessed. But to
generate the index file we need to unpack the crate and call
`cargo metadata`, which performs some sanity checks on the package
structure. The implication is that if #:skip-build? is used for
reasons other than performance - e.g., for a package that works
fine as a dependency but somehow fails during a standalone build -
then we might have trouble including such a package in Cargo's
local registry. I don't know how big an issue this is in practice,
but I think I encountered one package that fits this scenario:
custom_derive-0.1.7. (We can discuss the details in a separate
thread if you're interested.)
2) When building against a local-registry, it seems like Cargo
expects to find crates that it wouldn't for a vendor tree. As
mentioned above, my current patch pulls source crates into the
profile using the same logic as cargo-build-system currently does:
for each manifest package and its cargo-development-inputs,
recursively walk through all of the cargo-inputs. But after
building a profile this way, I encountered the following:
$ cargo build
error: no matching package named `automod` found
location searched: registry `crates-io`
required by package `serde_json v1.0.111`
... which satisfies dependency `serde_json = "^1.0"` of
package `rand_chacha v0.3.1`
... which satisfies dependency `rand_chacha = "^0.3.0"`
of package `rand v0.8.5`
... which satisfies dependency `rand = "^0.8.5"` of
package `mytest v0.1.0 (/home/guix/mytest)`
Note that serde_json is actually a dev-dependency of rand_chacha,
and automod is a dev-dependency of serde_json, but our algorithm
above assumes that dev-dependencies of dependencies can be
omitted. So in practice, it seems like building a local-registry
that cargo will accept is even more expensive than building a
vendor tree.
3) The build error above is interesting for another reason: cargo
simply gives up rather than fetching the missing dependencies. In
our earlier discussion, the motivation for using a local-registry
over a vendor tree in a profile was to support a sort of hybrid
workflow: the profile seeds the registry with some packages, but
cargo can still download others on its own during `cargo
build`. But in this test and several others, I haven't observed a
case where cargo falls back to crates.io for missing
packages. Efraim, could you share details on how you accomplished
this in the past? I wonder if our discussion might have conflated
cargo's registry cache ($CARGO_HOME/registry) with a true
local-registry; I haven't found any good documentation on the
former, but it has a different directory layout and index file
format from the local-registry's, so it wouldn't surprise me if it
had different semantics as well.
Cheers,
Jason
