Hello, I just submitted a patch inside the bitbake fetcher to display all the missing checksum at once instead of only one.
For the moment, I have no better solution, let's see what bitbake folks tell us about that. On Fri, 31 Mar 2023 at 07:32, Frederic Martinsons via lists.openembedded.org <frederic.martinsons=gmail....@lists.openembedded.org> wrote: > Hello, > > The multiple Cargo.lock seems totally valid, a project can have multiple > binaries provided (which is the case for Solana cli suite) so I'll submit a > new v6 of my series to reflect that, I just have to ignored specify dirs > like .git (no use to walk inside it) or .pc (can contains a patched > Cargo.lock that would be redundant and syntaxically invalid ) > > I'll also override the crate names with their version for all the crates > found. > > For the chicken and eggs issue, I need to think more of it (maybe make > bitbake print the list of all missing checksum would be enough) and, in > order to not delay this series, I'll submit a separate bitbake patch if I > manage to find something suitable. > > Many thanks for the comments Martin. > > Le jeu. 30 mars 2023, 23:15, Martin Jansa <martin.ja...@gmail.com> a > écrit : > >> I've added the updated .inc files to meta-webosose fork if it's easier to >> see there: >> >> https://github.com/shr-project/meta-webosose/commits/master-2023-03-30 >> (top 2 commits) >> >> this checksum mismatch is also the reason for that ugly exception as >> explained in top commit, it's race-condition between sugar.do_unpack and >> solana-cli.do_fetch where sugar has right arrayvec 0.7.2 with right >> checksum in do_unpack, while solana-cli renames it to _bad-checksum_ >> suffix, because solana-cli looks for checksum of arrayvec 0.7.1. >> >> I've tried to manually add name= param to verify this, but there are too >> many :), so I'll update the bbclass to always add name with version even if >> it doesn't see duplicated crates (and then this can be improved if multiple >> Cargo.lock files really need to be supported). >> >> I honestly don't know, I've was only working on these solana recipes for >> short time and wanted to offer them more as strange example. >> >> Cheers, >> >> On Thu, Mar 30, 2023 at 10:22 PM Martin Jansa via lists.openembedded.org >> <Martin.Jansa=gmail....@lists.openembedded.org> wrote: >> >>> On Thu, Mar 30, 2023 at 10:07 PM Frédéric Martinsons < >>> frederic.martins...@gmail.com> wrote: >>> >>>> By the way, about this "chickens and eggs" problem, isn't it the same >>>> for a regular recipe you just upgraded? >>>> >>> >>> I don't know, I haven't upgraded a recipe with crates yet :) >>> >>> It's just this time I was aware that the .inc file will need to be >>> refreshed and it wasn't obvious at first that easiest way to avoid do_fetch >>> issue blocking .inc refresh is to remove all crates first by emptying .inc >>> file. >>> >>> Personnaly, when I update a recipe, I let bitbake tell me what is the >>>> new checksum expected and put it the recipe (as the error message says). >>>> >>> >>> For regular recipes with a few checksum I do the same. But it doesn't >>> scale here as you mention bellow. >>> as it reported only the first wrong checksum from MANY. >>> >>> But I'm aware this is not exactly the same, since a cargo based recipe >>>> could contain a ton of crate:// uri and if you apply this method, you have >>>> to copy the new checksum one by one, bitbake error after another... >>>> >>>> Don't know how to make this better and I plead guilty for not >>>> mentioning that in a dedicated commit message. >>>> >>> >>> Better late than never, you can include it in v6 of this patch (I have >>> read the git log of the .bbclass before going to refresh the .inc files, so >>> others might notice it there as well). >>> >>> Of course, if someone come up with a smoothier way of doing this, I'll >>>> make a new patch. >>>> >>> >>> I think I found what the other issue with arrayvec create was: >>> >>> meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate:// >>> crates.io/arrayvec/0.7.2 \ >>> meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] >>> = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6" >>> meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate:// >>> crates.io/arrayvec/0.7.1 \ >>> meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] >>> = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd" >>> meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate:// >>> crates.io/arrayvec/0.7.2 \ >>> meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] >>> = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6" >>> >>> So different Cargo.lock files included in the same ${S} have different >>> version, but in this case it didn't set different name parameter, probably >>> because different Cargo.lock files are processed separately? >>> >>> Thanks >>> >>> Le jeu. 30 mars 2023, 21:50, Frédéric Martinsons < >>>> frederic.martins...@gmail.com> a écrit : >>>> >>>>> Well I see what you mean, I'll take a look at your example to try to >>>>> find out if multiple Cargo.lock could be expected. >>>>> >>>>> And for your second remark, yes, there is a chicken and eggs issue for >>>>> updating crates checksum from scratch. >>>>> You didn't miss anything, I came across this when updating crates >>>>> checksum that was "pristine" >>>>> and I only manage to execute update_crates by locally patching >>>>> bitbake/lib/bb/fetch2/crate.py for >>>>> recommends_checksum method to return False . >>>>> I'm aware that is not ideal but I don't know how to make this better >>>>> (maybe make do_update_crates >>>>> after do_fetch instead of after do_patch ? ) >>>>> >>>>> >>>>> Le jeu. 30 mars 2023, 21:23, Martin Jansa <martin.ja...@gmail.com> a >>>>> écrit : >>>>> >>>>>> I don't remember the exact details now, but when I was working on >>>>>> updating solana recipes to use this >>>>>> >>>>>> https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4 >>>>>> >>>>>> the S was pointing to just some subdirectory and multiple Cargo.locks >>>>>> files were parsed in other directories as well, that's why I was adding >>>>>> >>>>>> https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58 >>>>>> >>>>>> in the end I've changed to use the root directory in S and just >>>>>> set CARGO_SRC_DIR to right subdirectory to build. >>>>>> >>>>>> I don't have much experience with crates other than these solana >>>>>> recipes, but isn't there some valid use-case for multiple Cargo.locks? >>>>>> I assume Alex in original implementation didn't use the os.walk just >>>>>> to make it more complicated :). >>>>>> >>>>>> And FWIW when trying to regenerate these .inc files with current >>>>>> master and with: >>>>>> $ bitbake -c update_crates solana-keygen >>>>>> it fails with: >>>>>> ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for >>>>>> '/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please >>>>>> add at least one to the recipe: >>>>>> SRC_URI[Inflector.sha256sum] = >>>>>> "fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3" >>>>>> ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error: >>>>>> NoChecksumError('Missing SRC_URI checksum', ' >>>>>> https://crates.io/api/v1/crates/Inflector/0.11.4/download') >>>>>> >>>>>> Isn't it an chicken&egg issue now when do_fetch enforces checksums to >>>>>> be used? Or am I just missing some of the pending changes or just using >>>>>> it >>>>>> incorrectly? >>>>>> >>>>>> >>>>>> On Thu, Mar 30, 2023 at 6:34 PM Alex Kiernan <alex.kier...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Mar 30, 2023 at 4:45 PM <frederic.martins...@gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > From: Frederic Martinsons <frederic.martins...@gmail.com> >>>>>>> > >>>>>>> > There is no need to do such things, Cargo.lock file >>>>>>> > has to be at the root of CARGO_LOCK_SRC_DIR. >>>>>>> > This avoid finding other possible Cargo.lock that >>>>>>> > would be in subdir (for example if a patch is applied >>>>>>> > on the recipe, we can have .pc subdir in S and a Cargo.lock >>>>>>> > can be there) >>>>>>> > >>>>>>> > Signed-off-by: Frederic Martinsons <frederic.martins...@gmail.com> >>>>>>> > --- >>>>>>> > .../cargo-update-recipe-crates.bbclass | 12 >>>>>>> ++++++++---- >>>>>>> > 1 file changed, 8 insertions(+), 4 deletions(-) >>>>>>> > >>>>>>> > diff --git >>>>>>> a/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> b/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> > index daa363b0dd..549cfe627e 100644 >>>>>>> > --- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> > +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> > @@ -68,10 +68,14 @@ def get_crates(f): >>>>>>> > import os >>>>>>> > crates = "# Autogenerated with 'bitbake -c update_crates >>>>>>> ${PN}'\n\n" >>>>>>> > found = False >>>>>>> > -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): >>>>>>> > - for file in files: >>>>>>> > - if file == 'Cargo.lock': >>>>>>> > - crates += get_crates(os.path.join(root, file)) >>>>>>> > +for file in os.listdir('${CARGO_LOCK_SRC_DIR}'): >>>>>>> > + if file == 'Cargo.lock': >>>>>>> > + try: >>>>>>> > + cargo_lock_path = >>>>>>> os.path.join('${CARGO_LOCK_SRC_DIR}', file) >>>>>>> > + crates += get_crates(cargo_lock_path) >>>>>>> > + except Exception as e: >>>>>>> > + raise ValueError("Cannot parse '%s'" % >>>>>>> cargo_lock_path) from e >>>>>>> > + else: >>>>>>> > found = True >>>>>>> > if not found: >>>>>>> > raise ValueError("Unable to find Cargo.lock in >>>>>>> ${CARGO_LOCK_SRC_DIR}") >>>>>>> >>>>>>> Isn't this just a long-winded version of something like this >>>>>>> (completely untested): >>>>>>> >>>>>>> diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> b/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> index daa363b0dd65..22ddcfa5c1e3 100644 >>>>>>> --- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass >>>>>>> @@ -67,14 +67,7 @@ def get_crates(f): >>>>>>> >>>>>>> import os >>>>>>> crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" >>>>>>> -found = False >>>>>>> -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): >>>>>>> - for file in files: >>>>>>> - if file == 'Cargo.lock': >>>>>>> - crates += get_crates(os.path.join(root, file)) >>>>>>> - found = True >>>>>>> -if not found: >>>>>>> - raise ValueError("Unable to find Cargo.lock in >>>>>>> ${CARGO_LOCK_SRC_DIR}") >>>>>>> +crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", >>>>>>> "Cargo.lock")) >>>>>>> open("${TARGET_FILE}", 'w').write(crates) >>>>>>> EOF >>>>>>> >>>>>>> -- >>>>>>> Alex Kiernan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>> >>> >>> > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#179468): https://lists.openembedded.org/g/openembedded-core/message/179468 Mute This Topic: https://lists.openembedded.org/mt/97953740/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
