Am 12.02.2025 um 15:11 schrieb Bruce Ashfield:
On Wed, Feb 12, 2025 at 3:41 AM Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-...@weidmueller.com> wrote:
Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via
lists.openembedded.org <http://lists.openembedded.org>:
In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go,
npm and rust
on 11/02/2025 Stefan Herbrechtsmeier vialists.openembedded.org
<http://lists.openembedded.org> wrote:
From: Stefan Herbrechtsmeier<stefan.herbrechtsme...@weidmueller.com>
<mailto:stefan.herbrechtsme...@weidmueller.com>
The series adds on-the-fly support for package manager specific
dependencies and vendor directories. It contains the following changes:
1. Adds an early fetch, unpack and patch task to unpack and patch source
code with an embedded lock file for dependencies.
2. Parse the go.sum, Cargo.lock and package-lock.json lock files and
resolve the dependencies to SRC_URIs.
3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle
the SRC_URI files beside the SRC_URIs in the recipe.
I made a few comments, and will have another / better look at the
series tomorrow. There's a lot here, and it is hard to wrap my head
around everything that is changing.
I have one specific question below (from the point of view of go).
I've been looking through the series, and can't pick out where #3 is
done.
The files are created in the package manager specific vendor
classes (17 - 19) in the do_vendor_resolve task. The parsing is
implemented in the package manager specific modules in the vendor
package (6-8).
aha. I'll have a closer look in a bit!
I see patch 14 using SRC_URI_FILES, but where are those files
written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is
written to those files ?
The file is written into the vendor directory below WORKDIR. The
approach is similar to other classes which need to share data
between tasks.
The files contain individual SRC_URIs per line.
I may have missed that in the patches, but if that isn't described in
the commit logs, then
that description would be a nice addition. Along with a high level
description for each
supported language package manager about what is in the files and what
uses it to
resolve the dependencies.
I will extend the commit messages.
The concept that I'm not understanding (and that's just me not being
familiar with things, I'll continue reading the series) is that when
we suggested we'd like to have a mode where the dependencies could
clearly be listed in the SRC_URI, at least I was just thinking about a
way run the fetch/module elements that you were adding, write them to
a file and then have the recipe include it.
The additional SRC_URIs are dynamically added to the recipe
without the indirection over an include file. The generated
SRC_URIs could be inspected via the SRC_URI_FILES and manipulated
via patches for the lock files or an separated lock file.
The issue with patching those dynamic files .. is just that, they are
dynamic and
they could be regenerated (but hopefully without a SRCREV change they
are identically
generated each time, so maybe it isn't a big deal) and patching could
fail. Much like we
don't patch generated automake files, we patch the .am and .in files
that generate the
outputs.
You can't compare lock files with automake files. The lock files is
generated but it contains additional information. The only difference is
that a program and not a human write the file. If an update creates too
much noise in the file we should discuss with upstream how to minimize
it. The lock file is designed to be under revision control and
shouldn't change without reason. Dependabot automatically creates
updates and pull requests for dependency updates. This will not work if
the lock file isn't deterministic.
I can't tell if in the series those files are written each time, and
that there would be no way to edit those SRC_URI_FILES .. but I'll
look again tomorrow.
The files are written if the do_vendor_resolve task need to be
rerun. It isn't possible to edit the generated SRC_URIs. Instead
the lock file could be patched.
ok. So they are really just a way to dump what is being built to disk
and out of "memory",
it really isn't what I was looking for, but at least I understand what
you were trying to
do with the writing and later use of the files now.
I was hoping for something that I could include "statically" (for lack
of a better term) in my
recipes, and then have the handlers for the protocols do the fetching
.. again, much like
the crate:// fetches in the .inc files.
I don't use the fetcher because of its drawbacks and limitations. I use
the same wget URL without the indirection over the package manager
specific fetchers. I can follow the pypi approach to generate SRC_URIs
which call a python function:
${@crate_src_uri(d, 'xyz', '1.2.3')}
But why you need this? In any case the manipulation of a dependency
could lead to interdependence between dependencies.
To be clear, I'm not suggesting that mode would be the only mode, just
that the way the
generation of the fetches -> actual fetching was constructed such that
I could avoid
"short circuit" the handling of the lock file and simply list the
dependencies for fetching
(much like how I can go get all of the SRCREVS for a gitsm:// fetch,
and put them into
my recipes as pure git:// fetches if I'm having issues with the git
submodles).
Add a function to create this file isn't a problem, but I can't ensure
that this will work in any case and doesn't create problems because of
the discrepancy between the manifest / lock file and the available sources.
That file would manipulate the standard SRC_URI. In other words still
support a mode that is like the .inc files with crate://. So someone
could either have the lockfile parsed and fetched, or have a way to
run the parsing and fetching via a task, write a file and include the
file in their recipe to short circuit the processing of the lockfile.
(meaning the expanded and end fetches that are done once you've
processed the file are simply listed as a series of fetches that are
carried out without extra processing .. and "unrolled" dependency
file pointing at the "sources" git, crate, mod, whatever)
What is the motivation "to short circuit the processing of the
lockfile"? Like any task the processing and its output are cached
via the sstate cache.
See above. It isn't about re-running it a later time and re-using the
artifacts, it is about
having a way to make sure nothing is dynamically resolved when bitbake
<foo> is called,
just fetch and the rest of the standard tasks.
The resolve always create the same URL like the crate fetcher. It only
changes if you change the resolve function or the crate fetcher. Every
needed information is defined inside the lock file.
It is possible to use the vendor package to create a include file
but we should have good reasons to support two solutions for the
same problem.
If that just doesn't make sense, then if there was a way to copy
the lockfile out of the recipe and have it overlayed onto the fetched
one .. maybe breaking out the individual fetch lines isn't required,
since they could be individually manipulated in that lockfile.
I haven't test it but this should be possible. You can place a
lock file beside the recipe, add it to the SRC_URI and change the
XXX_LOCK_PATH.
Something like that is a big step towards having that finer grained
control in a recipe that
I'm looking for. I'd still like some sort of line by line "dumped
dependency list" if there's a
lower level protocol at play (but the lock file really also is just
that line by line list for many
of the languages).
Why you need this finer grained control in the recipe? You have the
finer grained control in the lock file and additional tools to manage
the file and all its interdependence. The resolve is deterministic and
the .inc is only another representation. Manipulation of the
representation could lead to problems and complicate the process. It's
like you translate the .am and .in files into a .inc file to influence
the generated automake files.
Would it help if we have a devtool shell so that you can run "cargo
update" inside the folder created by "devtool modify" to update
dependencies via the common tools and create a patch?
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#211305):
https://lists.openembedded.org/g/openembedded-core/message/211305
Mute This Topic: https://lists.openembedded.org/mt/111133313/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
