On Wed, Feb 12, 2025 at 3:41 AM Stefan Herbrechtsmeier < stefan.herbrechtsmeier-...@weidmueller.com> wrote:
> Am 12.02.2025 um 00:14 schrieb Bruce Ashfield via lists.openembedded.org: > > In message: [bitbake-devel] [RFC PATCH 00/30] Add vendor support for go, npm > and rust > on 11/02/2025 Stefan Herbrechtsmeier via lists.openembedded.org wrote: > > > From: Stefan Herbrechtsmeier <stefan.herbrechtsme...@weidmueller.com> > <stefan.herbrechtsme...@weidmueller.com> > > The series adds on-the-fly support for package manager specific > dependencies and vendor directories. It contains the following changes: > 1. Adds an early fetch, unpack and patch task to unpack and patch source > code with an embedded lock file for dependencies. > 2. Parse the go.sum, Cargo.lock and package-lock.json lock files and > resolve the dependencies to SRC_URIs. > 3. Save the SRC_URIs in a file and adapt all SRC_URIs users to handle > the SRC_URI files beside the SRC_URIs in the recipe. > > I made a few comments, and will have another / better look at the > series tomorrow. There's a lot here, and it is hard to wrap my head > around everything that is changing. > > I have one specific question below (from the point of view of go). > > I've been looking through the series, and can't pick out where #3 is > done. > > The files are created in the package manager specific vendor classes (17 - > 19) in the do_vendor_resolve task. The parsing is implemented in the > package manager specific modules in the vendor package (6-8). > > aha. I'll have a closer look in a bit! > I see patch 14 using SRC_URI_FILES, but where are those files > written ? Is that in patch 18 (vendor_go_do_vendor_resolve ?) What is > written to those files ? > > The file is written into the vendor directory below WORKDIR. The approach > is similar to other classes which need to share data between tasks. > > The files contain individual SRC_URIs per line. > I may have missed that in the patches, but if that isn't described in the commit logs, then that description would be a nice addition. Along with a high level description for each supported language package manager about what is in the files and what uses it to resolve the dependencies. > The concept that I'm not understanding (and that's just me not being > familiar with things, I'll continue reading the series) is that when > we suggested we'd like to have a mode where the dependencies could > clearly be listed in the SRC_URI, at least I was just thinking about a > way run the fetch/module elements that you were adding, write them to > a file and then have the recipe include it. > > The additional SRC_URIs are dynamically added to the recipe without the > indirection over an include file. The generated SRC_URIs could be inspected > via the SRC_URI_FILES and manipulated via patches for the lock files or an > separated lock file. > The issue with patching those dynamic files .. is just that, they are dynamic and they could be regenerated (but hopefully without a SRCREV change they are identically generated each time, so maybe it isn't a big deal) and patching could fail. Much like we don't patch generated automake files, we patch the .am and .in files that generate the outputs. > > I can't tell if in the series those files are written each time, and > that there would be no way to edit those SRC_URI_FILES .. but I'll > look again tomorrow. > > The files are written if the do_vendor_resolve task need to be rerun. It > isn't possible to edit the generated SRC_URIs. Instead the lock file could > be patched. > ok. So they are really just a way to dump what is being built to disk and out of "memory", it really isn't what I was looking for, but at least I understand what you were trying to do with the writing and later use of the files now. I was hoping for something that I could include "statically" (for lack of a better term) in my recipes, and then have the handlers for the protocols do the fetching .. again, much like the crate:// fetches in the .inc files. To be clear, I'm not suggesting that mode would be the only mode, just that the way the generation of the fetches -> actual fetching was constructed such that I could avoid "short circuit" the handling of the lock file and simply list the dependencies for fetching (much like how I can go get all of the SRCREVS for a gitsm:// fetch, and put them into my recipes as pure git:// fetches if I'm having issues with the git submodles). That file would manipulate the standard SRC_URI. In other words still > support a mode that is like the .inc files with crate://. So someone > could either have the lockfile parsed and fetched, or have a way to > run the parsing and fetching via a task, write a file and include the > file in their recipe to short circuit the processing of the lockfile. > (meaning the expanded and end fetches that are done once you've > processed the file are simply listed as a series of fetches that are > carried out without extra processing .. and "unrolled" dependency > file pointing at the "sources" git, crate, mod, whatever) > > What is the motivation "to short circuit the processing of the lockfile"? > Like any task the processing and its output are cached via the sstate cache. > See above. It isn't about re-running it a later time and re-using the artifacts, it is about having a way to make sure nothing is dynamically resolved when bitbake <foo> is called, just fetch and the rest of the standard tasks. > > It is possible to use the vendor package to create a include file but we > should have good reasons to support two solutions for the same problem. > > If that just doesn't make sense, then if there was a way to copy > the lockfile out of the recipe and have it overlayed onto the fetched > one .. maybe breaking out the individual fetch lines isn't required, > since they could be individually manipulated in that lockfile. > > I haven't test it but this should be possible. You can place a lock file > beside the recipe, add it to the SRC_URI and change the XXX_LOCK_PATH. > Something like that is a big step towards having that finer grained control in a recipe that I'm looking for. I'd still like some sort of line by line "dumped dependency list" if there's a lower level protocol at play (but the lock file really also is just that line by line list for many of the languages). Bruce > Regards Stefan > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#211233): https://lists.openembedded.org/g/openembedded-core/message/211233 Mute This Topic: https://lists.openembedded.org/mt/111133313/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
